2024-03-01 12:53:17,595 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-01 12:53:17,936 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-01 12:53:18,056 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-01 12:53:18,122 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-01 12:53:18,132 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-01 12:53:18,150 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=274, ProcessCount=171, AvailableMemoryMB=10609
2024-03-01 12:53:18,152 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-01 12:53:18,153 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-01 12:53:18,297 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026012101:84 , srv1036331837:84 , srv1054283001:84 , srv1069793759:84 , srv1071615532:84 , srv1086415:84 , srv1108745995:84 , srv1137007330:84 , srv1201097753:84 , srv1205869824:84 , srv1211473304:84 , srv1223889577:84 , srv1254464242:84 , srv1274217533:84 , srv1283249812:84 , srv1300967460:84 , srv1328361299:84 , srv132855945:84 , srv134758258:84 , srv1350179908:84 , srv1352218352:84 , srv1370568293:84 , srv1389253603:84 , srv140680801:84 , srv1429066262:84 , srv1460550654:84 , srv1464112739:84 , srv1466630446:84 , srv1468122706:84 , srv1511758567:84 , srv1551435071:84 , srv1580047667:84 , srv1593055697:84 , srv160756848:84 , srv1624816734:84 , srv1628313014:84 , srv1636442060:84 , srv1642540118:84 , srv1647175964:84 , srv1649651363:84 , srv1659600541:84 , srv1666838767:84 , srv1692940929:84 , srv17283885:84 , srv1734663898:84 , srv1764826202:84 , srv1794468048:84 , srv1799624868:84 , srv180098678:84 , srv1838304457:84 , srv1845908376:84 , srv1848332710:84 , srv1879025357:84 , srv188565480:84 , srv1940483854:84 , srv1999137180:84 , srv2004856143:84 , srv2028757447:84 , srv204107055:84 , srv205692560:84 , srv2108442362:84 , srv2114266643:84 , srv2137685854:84 , srv240573768:84 , srv254381240:84 , srv295405867:84 , srv327537199:84 , srv334937685:84 , srv367091719:84 , srv402376795:84 , srv416155699:84 , srv44029806:84 , srv451592021:84 , srv455974126:84 , srv462398044:84 , srv463030025:84 , srv485187435:84 , srv514447207:84 , srv528638802:84 , srv538337901:84 , srv546814221:84 , srv578025858:84 , srv579804826:84 , srv635600664:84 , srv642725057:84 , srv667872149:84 , srv710666253:84 , srv766219667:84 , srv805036295:84 , srv821888787:84 , srv863862794:84 , srv880813904:84 , srv890260389:84 , srv913746800:84 , srv949105195:84 , srv952992063:84 , srv954659345:84 , srv962780576:84 , srv984860820:84 , srv1313720909:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-01 12:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv984860820=99, srv1845908376=51, srv1370568293=22, srv1352218352=21, srv1211473304=10, srv240573768=64, srv17283885=44, srv1848332710=52, srv1026012101=0, srv821888787=90, srv402376795=70, srv334937685=68, srv485187435=77, srv1464112739=27, srv132855945=18, srv949105195=95, srv1460550654=26, srv667872149=86, srv528638802=79, srv1036331837=1, srv1054283001=2, srv1071615532=4, srv880813904=92, srv1551435071=31, srv1254464242=12, srv863862794=91, srv952992063=96, srv1879025357=53, srv455974126=74, srv367091719=69, srv1429066262=25, srv1692940929=43, srv327537199=67, srv538337901=80, srv1940483854=55, srv1659600541=41, srv766219667=88, srv1838304457=50, srv2137685854=63, srv1205869824=9, srv1999137180=56, srv1086415=5, srv2004856143=57, srv463030025=76, srv1511758567=30, srv1593055697=33, srv295405867=66, srv1137007330=7, srv1300967460=15, srv1328361299=17, srv1764826202=46, srv1624816734=35, srv546814221=81, srv1636442060=37, srv954659345=97, srv1108745995=6, srv1313720909=16, srv462398044=75, srv416155699=71, srv1350179908=20, srv1642540118=38, srv514447207=78, srv451592021=73, srv1201097753=8, srv44029806=72, srv134758258=19, srv642725057=85, srv2108442362=61, srv890260389=93, srv635600664=84, srv188565480=54, srv1466630446=28, srv1223889577=11, srv140680801=24, srv710666253=87, srv1649651363=40, srv204107055=59, srv1628313014=36, srv2114266643=62, srv254381240=65, srv1580047667=32, srv180098678=49, srv160756848=34, srv579804826=83, srv962780576=98, srv1069793759=3, srv913746800=94, srv1468122706=29, srv1283249812=14, srv1274217533=13, srv805036295=89, srv1794468048=47, srv578025858=82, srv1389253603=23, srv1647175964=39, srv1799624868=48, srv1734663898=45, srv2028757447=58, srv1666838767=42, srv205692560=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:53:18,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-01 12:53:18,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-01 12:53:18,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-01 12:53:18,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-01 12:53:18,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-01 12:53:18,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-01 12:53:18,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-01 12:53:18,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-01 12:53:18,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-01 12:53:18,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-01 12:53:18,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-01 12:53:18,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-01 12:53:18,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-01 12:53:18,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-01 12:53:18,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-01 12:53:18,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-01 12:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-01 12:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-01 12:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-01 12:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-01 12:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-01 12:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-01 12:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-01 12:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-01 12:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-01 12:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-01 12:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-01 12:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-01 12:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-01 12:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-01 12:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-01 12:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-01 12:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-01 12:53:18,635 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:18,635 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-01 12:53:18,635 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-01 12:53:19,003 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 529 ms to try 5000 different iterations.  Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17733276278441093. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04105480416170809, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05894127648833263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04105480416170809, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23758333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:19,025 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2004856143:81 , srv890260389:81 , srv1799624868:82 , srv188565480:82 , srv485187435:82 , srv821888787:82 , srv1370568293:83 , srv1879025357:83 , srv635600664:83 , srv710666253:83 , srv1069793759:84 , srv1274217533:84 , srv1460550654:84 , srv1593055697:84 , srv1642540118:84 , srv180098678:84 , srv1838304457:84 , srv1999137180:84 , srv254381240:84 , srv455974126:84 , srv463030025:84 , srv667872149:84 , srv863862794:84 , srv1205869824:85 , srv1223889577:85 , srv1628313014:85 , srv1636442060:85 , srv1692940929:85 , srv204107055:85 , srv205692560:85 , srv462398044:85 , srv528638802:85 , srv546814221:85 , srv954659345:85 , srv962780576:85 , srv984860820:85 , srv1054283001:86 , srv1108745995:86 , srv132855945:86 , srv134758258:86 , srv1350179908:86 , srv1464112739:86 , srv1466630446:86 , srv1468122706:86 , srv1666838767:86 , srv1845908376:86 , srv1940483854:86 , srv2028757447:86 , srv2114266643:86 , srv367091719:86 , srv451592021:86 , srv538337901:86 , srv578025858:86 , srv579804826:86 , srv1026012101:87 , srv1071615532:87 , srv1137007330:87 , srv1300967460:87 , srv1352218352:87 , srv1429066262:87 , srv1511758567:87 , srv1580047667:87 , srv1647175964:87 , srv1734663898:87 , srv1764826202:87 , srv2108442362:87 , srv295405867:87 , srv327537199:87 , srv880813904:87 , srv952992063:87 , srv1036331837:88 , srv1211473304:88 , srv1254464242:88 , srv1389253603:88 , srv160756848:88 , srv1649651363:88 , srv1659600541:88 , srv17283885:88 , srv1794468048:88 , srv2137685854:88 , srv240573768:88 , srv642725057:88 , srv805036295:88 , srv913746800:88 , srv949105195:88 , srv1086415:89 , srv1283249812:89 , srv1328361299:89 , srv1624816734:89 , srv514447207:89 , srv766219667:89 , srv1201097753:90 , srv334937685:90 , srv140680801:91 , srv402376795:91 , srv416155699:91 , srv1551435071:93 , srv1848332710:93 , srv44029806:93 , srv1313720909:455 }
2024-03-01 12:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv984860820=99, srv1845908376=51, srv1370568293=22, srv1352218352=21, srv1211473304=10, srv240573768=64, srv17283885=44, srv1848332710=52, srv1026012101=0, srv821888787=90, srv402376795=70, srv334937685=68, srv485187435=77, srv1464112739=27, srv132855945=18, srv949105195=95, srv1460550654=26, srv667872149=86, srv528638802=79, srv1036331837=1, srv1054283001=2, srv1071615532=4, srv880813904=92, srv1551435071=31, srv1254464242=12, srv863862794=91, srv952992063=96, srv1879025357=53, srv455974126=74, srv367091719=69, srv1429066262=25, srv1692940929=43, srv327537199=67, srv538337901=80, srv1940483854=55, srv1659600541=41, srv766219667=88, srv1838304457=50, srv2137685854=63, srv1205869824=9, srv1999137180=56, srv1086415=5, srv2004856143=57, srv463030025=76, srv1511758567=30, srv1593055697=33, srv295405867=66, srv1137007330=7, srv1300967460=15, srv1328361299=17, srv1764826202=46, srv1624816734=35, srv546814221=81, srv1636442060=37, srv954659345=97, srv1108745995=6, srv1313720909=16, srv462398044=75, srv416155699=71, srv1350179908=20, srv1642540118=38, srv514447207=78, srv451592021=73, srv1201097753=8, srv44029806=72, srv134758258=19, srv642725057=85, srv2108442362=61, srv890260389=93, srv635600664=84, srv188565480=54, srv1466630446=28, srv1223889577=11, srv140680801=24, srv710666253=87, srv1649651363=40, srv204107055=59, srv1628313014=36, srv2114266643=62, srv254381240=65, srv1580047667=32, srv180098678=49, srv160756848=34, srv579804826=83, srv962780576=98, srv1069793759=3, srv913746800=94, srv1468122706=29, srv1283249812=14, srv1274217533=13, srv805036295=89, srv1794468048=47, srv578025858=82, srv1389253603=23, srv1647175964=39, srv1799624868=48, srv1734663898=45, srv2028757447=58, srv1666838767=42, srv205692560=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:53:19,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:53:19,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:53:19,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:53:19,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-01 12:53:19,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-01 12:53:19,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-01 12:53:19,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-01 12:53:19,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-01 12:53:19,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-01 12:53:19,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-01 12:53:19,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-01 12:53:19,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-01 12:53:19,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-01 12:53:19,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-01 12:53:19,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-01 12:53:19,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04105480416170809, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05894127648833263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04105480416170809, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23758333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:19,212 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-01 12:53:19,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17729984567885954, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04105480416170809, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05894127648833263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04105480416170809, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23758333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-01 12:53:19,552 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 472 ms to try 5000 different iterations.  Found a solution that moves 2590 regions; Going from a computed imbalance of 0.17729984567885954 to a new imbalance of 0.015450131947975697. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02114033065604494); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02114033065604494); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:19,564 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1551435071:78 , srv44029806:79 , srv140680801:80 , srv416155699:80 , srv1201097753:81 , srv402376795:81 , srv1086415:82 , srv1848332710:82 , srv766219667:82 , srv1328361299:83 , srv1389253603:83 , srv1649651363:83 , srv514447207:83 , srv642725057:83 , srv913746800:83 , srv1036331837:84 , srv1211473304:84 , srv1352218352:84 , srv160756848:84 , srv1624816734:84 , srv1734663898:84 , srv1794468048:84 , srv805036295:84 , srv880813904:84 , srv952992063:84 , srv1026012101:85 , srv1071615532:85 , srv1254464242:85 , srv1283249812:85 , srv1350179908:85 , srv1464112739:85 , srv1466630446:85 , srv1468122706:85 , srv1511758567:85 , srv1659600541:85 , srv2108442362:85 , srv2137685854:85 , srv240573768:85 , srv295405867:85 , srv327537199:85 , srv334937685:85 , srv546814221:85 , srv1054283001:86 , srv1137007330:86 , srv1300967460:86 , srv1628313014:86 , srv1636442060:86 , srv1647175964:86 , srv1764826202:86 , srv1838304457:86 , srv1845908376:86 , srv204107055:86 , srv205692560:86 , srv254381240:86 , srv451592021:86 , srv528638802:86 , srv538337901:86 , srv578025858:86 , srv579804826:86 , srv949105195:86 , srv962780576:86 , srv1205869824:87 , srv1223889577:87 , srv132855945:87 , srv134758258:87 , srv1429066262:87 , srv1580047667:87 , srv1593055697:87 , srv1692940929:87 , srv17283885:87 , srv1999137180:87 , srv2028757447:87 , srv2114266643:87 , srv367091719:87 , srv455974126:87 , srv462398044:87 , srv463030025:87 , srv954659345:87 , srv984860820:87 , srv1069793759:88 , srv1274217533:88 , srv1460550654:88 , srv1642540118:88 , srv1666838767:88 , srv180098678:88 , srv1940483854:88 , srv635600664:88 , srv863862794:88 , srv1108745995:89 , srv1370568293:89 , srv710666253:89 , srv1799624868:90 , srv2004856143:90 , srv485187435:90 , srv667872149:90 , srv890260389:90 , srv1879025357:92 , srv188565480:92 , srv821888787:92 , srv1313720909:507 }
2024-03-01 12:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv984860820=99, srv1845908376=51, srv1370568293=22, srv1352218352=21, srv1211473304=10, srv240573768=64, srv17283885=44, srv1848332710=52, srv1026012101=0, srv821888787=90, srv402376795=70, srv334937685=68, srv485187435=77, srv1464112739=27, srv132855945=18, srv949105195=95, srv1460550654=26, srv667872149=86, srv528638802=79, srv1036331837=1, srv1054283001=2, srv1071615532=4, srv880813904=92, srv1551435071=31, srv1254464242=12, srv863862794=91, srv952992063=96, srv1879025357=53, srv455974126=74, srv367091719=69, srv1429066262=25, srv1692940929=43, srv327537199=67, srv538337901=80, srv1940483854=55, srv1659600541=41, srv766219667=88, srv1838304457=50, srv2137685854=63, srv1205869824=9, srv1999137180=56, srv1086415=5, srv2004856143=57, srv463030025=76, srv1511758567=30, srv1593055697=33, srv295405867=66, srv1137007330=7, srv1300967460=15, srv1328361299=17, srv1764826202=46, srv1624816734=35, srv546814221=81, srv1636442060=37, srv954659345=97, srv1108745995=6, srv1313720909=16, srv462398044=75, srv416155699=71, srv1350179908=20, srv1642540118=38, srv514447207=78, srv451592021=73, srv1201097753=8, srv44029806=72, srv134758258=19, srv642725057=85, srv2108442362=61, srv890260389=93, srv635600664=84, srv188565480=54, srv1466630446=28, srv1223889577=11, srv140680801=24, srv710666253=87, srv1649651363=40, srv204107055=59, srv1628313014=36, srv2114266643=62, srv254381240=65, srv1580047667=32, srv180098678=49, srv160756848=34, srv579804826=83, srv962780576=98, srv1069793759=3, srv913746800=94, srv1468122706=29, srv1283249812=14, srv1274217533=13, srv805036295=89, srv1794468048=47, srv578025858=82, srv1389253603=23, srv1647175964=39, srv1799624868=48, srv1734663898=45, srv2028757447=58, srv1666838767=42, srv205692560=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:53:19,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:53:19,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:53:19,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:53:19,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:53:19,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:53:19,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:53:19,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-01 12:53:19,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-01 12:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-01 12:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-01 12:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-01 12:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-01 12:53:19,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-01 12:53:19,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-01 12:53:19,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-01 12:53:19,722 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02114033065604494); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02114033065604494); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:19,722 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-01 12:53:19,723 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015431996380356437, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02114033065604494); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02114033065604494); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-01 12:53:20,031 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 415 ms to try 5000 different iterations.  Found a solution that moves 964 regions; Going from a computed imbalance of 0.015431996380356437 to a new imbalance of 6.737476610233955E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004647440486997607); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009870087677015036); MoveCostFunction : (multiplier=7.0, imbalance=0.10711111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004647440486997607); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:20,035 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1108745995:82 , srv1848332710:82 , srv1283249812:83 , srv1580047667:83 , srv1666838767:83 , srv17283885:83 , srv188565480:83 , srv334937685:83 , srv667872149:83 , srv1254464242:84 , srv132855945:84 , srv1429066262:84 , srv1647175964:84 , srv1764826202:84 , srv1879025357:84 , srv1940483854:84 , srv240573768:84 , srv821888787:84 , srv949105195:84 , srv1026012101:85 , srv1054283001:85 , srv1069793759:85 , srv1137007330:85 , srv1211473304:85 , srv1300967460:85 , srv134758258:85 , srv1370568293:85 , srv1460550654:85 , srv160756848:85 , srv1624816734:85 , srv1642540118:85 , srv1659600541:85 , srv1794468048:85 , srv1845908376:85 , srv2028757447:85 , srv2108442362:85 , srv2114266643:85 , srv2137685854:85 , srv327537199:85 , srv367091719:85 , srv44029806:85 , srv485187435:85 , srv538337901:85 , srv578025858:85 , srv579804826:85 , srv805036295:85 , srv1036331837:86 , srv1071615532:86 , srv1201097753:86 , srv1205869824:86 , srv1223889577:86 , srv1274217533:86 , srv1328361299:86 , srv140680801:86 , srv1464112739:86 , srv1466630446:86 , srv1628313014:86 , srv1636442060:86 , srv1649651363:86 , srv1692940929:86 , srv1734663898:86 , srv1799624868:86 , srv180098678:86 , srv1999137180:86 , srv205692560:86 , srv295405867:86 , srv402376795:86 , srv451592021:86 , srv462398044:86 , srv463030025:86 , srv514447207:86 , srv528638802:86 , srv635600664:86 , srv863862794:86 , srv880813904:86 , srv954659345:86 , srv984860820:86 , srv1350179908:87 , srv1352218352:87 , srv1389253603:87 , srv1468122706:87 , srv1511758567:87 , srv1551435071:87 , srv1593055697:87 , srv2004856143:87 , srv254381240:87 , srv416155699:87 , srv455974126:87 , srv546814221:87 , srv642725057:87 , srv710666253:87 , srv766219667:87 , srv890260389:87 , srv913746800:87 , srv952992063:87 , srv962780576:87 , srv1086415:88 , srv1838304457:88 , srv204107055:88 , srv1313720909:537 }
2024-03-01 12:53:20,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:53:20,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv984860820=99, srv1845908376=51, srv1370568293=22, srv1352218352=21, srv1211473304=10, srv240573768=64, srv17283885=44, srv1848332710=52, srv1026012101=0, srv821888787=90, srv402376795=70, srv334937685=68, srv485187435=77, srv1464112739=27, srv132855945=18, srv949105195=95, srv1460550654=26, srv667872149=86, srv528638802=79, srv1036331837=1, srv1054283001=2, srv1071615532=4, srv880813904=92, srv1551435071=31, srv1254464242=12, srv863862794=91, srv952992063=96, srv1879025357=53, srv455974126=74, srv367091719=69, srv1429066262=25, srv1692940929=43, srv327537199=67, srv538337901=80, srv1940483854=55, srv1659600541=41, srv766219667=88, srv1838304457=50, srv2137685854=63, srv1205869824=9, srv1999137180=56, srv1086415=5, srv2004856143=57, srv463030025=76, srv1511758567=30, srv1593055697=33, srv295405867=66, srv1137007330=7, srv1300967460=15, srv1328361299=17, srv1764826202=46, srv1624816734=35, srv546814221=81, srv1636442060=37, srv954659345=97, srv1108745995=6, srv1313720909=16, srv462398044=75, srv416155699=71, srv1350179908=20, srv1642540118=38, srv514447207=78, srv451592021=73, srv1201097753=8, srv44029806=72, srv134758258=19, srv642725057=85, srv2108442362=61, srv890260389=93, srv635600664=84, srv188565480=54, srv1466630446=28, srv1223889577=11, srv140680801=24, srv710666253=87, srv1649651363=40, srv204107055=59, srv1628313014=36, srv2114266643=62, srv254381240=65, srv1580047667=32, srv180098678=49, srv160756848=34, srv579804826=83, srv962780576=98, srv1069793759=3, srv913746800=94, srv1468122706=29, srv1283249812=14, srv1274217533=13, srv805036295=89, srv1794468048=47, srv578025858=82, srv1389253603=23, srv1647175964=39, srv1799624868=48, srv1734663898=45, srv2028757447=58, srv1666838767=42, srv205692560=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-01 12:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-01 12:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-01 12:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-01 12:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-01 12:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-01 12:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-01 12:53:20,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-01 12:53:20,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-01 12:53:20,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-01 12:53:20,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-01 12:53:20,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-01 12:53:20,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-01 12:53:20,228 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004647440486997607); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009870087677015036); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004647440486997607); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:20,228 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-01 12:53:20,229 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.669975887512078E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004647440486997607); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009870087677015036); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004647440486997607); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-01 12:53:20,671 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 584 ms to try 5000 different iterations.  Found a solution that moves 263 regions; Going from a computed imbalance of 6.669975887512078E-4 to a new imbalance of 2.404299288888515E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026801008406912323); MoveCostFunction : (multiplier=7.0, imbalance=0.029222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:20,672 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1879025357:82 , srv1071615532:83 , srv1086415:83 , srv1511758567:83 , srv1624816734:83 , srv1692940929:83 , srv1799624868:83 , srv1026012101:84 , srv1036331837:84 , srv1137007330:84 , srv1201097753:84 , srv1205869824:84 , srv1223889577:84 , srv1274217533:84 , srv1300967460:84 , srv1328361299:84 , srv134758258:84 , srv1350179908:84 , srv1352218352:84 , srv1389253603:84 , srv140680801:84 , srv1429066262:84 , srv1468122706:84 , srv1551435071:84 , srv1580047667:84 , srv1593055697:84 , srv1628313014:84 , srv1649651363:84 , srv1659600541:84 , srv17283885:84 , srv180098678:84 , srv1940483854:84 , srv2004856143:84 , srv204107055:84 , srv2114266643:84 , srv2137685854:84 , srv334937685:84 , srv367091719:84 , srv402376795:84 , srv416155699:84 , srv451592021:84 , srv455974126:84 , srv462398044:84 , srv463030025:84 , srv514447207:84 , srv642725057:84 , srv710666253:84 , srv766219667:84 , srv821888787:84 , srv863862794:84 , srv890260389:84 , srv913746800:84 , srv949105195:84 , srv952992063:84 , srv954659345:84 , srv962780576:84 , srv984860820:84 , srv1054283001:85 , srv1069793759:85 , srv1211473304:85 , srv1254464242:85 , srv1283249812:85 , srv1370568293:85 , srv1464112739:85 , srv1466630446:85 , srv160756848:85 , srv1636442060:85 , srv1642540118:85 , srv1647175964:85 , srv1666838767:85 , srv1794468048:85 , srv1838304457:85 , srv1845908376:85 , srv1848332710:85 , srv1999137180:85 , srv2028757447:85 , srv205692560:85 , srv2108442362:85 , srv240573768:85 , srv254381240:85 , srv295405867:85 , srv327537199:85 , srv44029806:85 , srv485187435:85 , srv528638802:85 , srv538337901:85 , srv546814221:85 , srv578025858:85 , srv579804826:85 , srv635600664:85 , srv805036295:85 , srv880813904:85 , srv1108745995:86 , srv132855945:86 , srv1460550654:86 , srv1734663898:86 , srv1764826202:86 , srv188565480:86 , srv667872149:86 , srv1313720909:643 }
2024-03-01 12:53:20,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:53:20,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv984860820=99, srv1845908376=51, srv1370568293=22, srv1352218352=21, srv1211473304=10, srv240573768=64, srv17283885=44, srv1848332710=52, srv1026012101=0, srv821888787=90, srv402376795=70, srv334937685=68, srv485187435=77, srv1464112739=27, srv132855945=18, srv949105195=95, srv1460550654=26, srv667872149=86, srv528638802=79, srv1036331837=1, srv1054283001=2, srv1071615532=4, srv880813904=92, srv1551435071=31, srv1254464242=12, srv863862794=91, srv952992063=96, srv1879025357=53, srv455974126=74, srv367091719=69, srv1429066262=25, srv1692940929=43, srv327537199=67, srv538337901=80, srv1940483854=55, srv1659600541=41, srv766219667=88, srv1838304457=50, srv2137685854=63, srv1205869824=9, srv1999137180=56, srv1086415=5, srv2004856143=57, srv463030025=76, srv1511758567=30, srv1593055697=33, srv295405867=66, srv1137007330=7, srv1300967460=15, srv1328361299=17, srv1764826202=46, srv1624816734=35, srv546814221=81, srv1636442060=37, srv954659345=97, srv1108745995=6, srv1313720909=16, srv462398044=75, srv416155699=71, srv1350179908=20, srv1642540118=38, srv514447207=78, srv451592021=73, srv1201097753=8, srv44029806=72, srv134758258=19, srv642725057=85, srv2108442362=61, srv890260389=93, srv635600664=84, srv188565480=54, srv1466630446=28, srv1223889577=11, srv140680801=24, srv710666253=87, srv1649651363=40, srv204107055=59, srv1628313014=36, srv2114266643=62, srv254381240=65, srv1580047667=32, srv180098678=49, srv160756848=34, srv579804826=83, srv962780576=98, srv1069793759=3, srv913746800=94, srv1468122706=29, srv1283249812=14, srv1274217533=13, srv805036295=89, srv1794468048=47, srv578025858=82, srv1389253603=23, srv1647175964=39, srv1799624868=48, srv1734663898=45, srv2028757447=58, srv1666838767=42, srv205692560=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:53:20,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:53:20,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:53:20,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:53:20,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:53:20,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:53:20,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-01 12:53:20,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-01 12:53:20,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-01 12:53:20,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-01 12:53:20,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-01 12:53:20,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-01 12:53:20,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-01 12:53:20,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-01 12:53:20,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-01 12:53:20,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-01 12:53:20,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-01 12:53:20,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-01 12:53:20,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-01 12:53:20,865 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.2201427528319458E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026801008406912323); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:20,865 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1879025357:82 , srv1071615532:83 , srv1086415:83 , srv1511758567:83 , srv1624816734:83 , srv1692940929:83 , srv1799624868:83 , srv1026012101:84 , srv1036331837:84 , srv1137007330:84 , srv1201097753:84 , srv1205869824:84 , srv1223889577:84 , srv1274217533:84 , srv1300967460:84 , srv1328361299:84 , srv134758258:84 , srv1350179908:84 , srv1352218352:84 , srv1389253603:84 , srv140680801:84 , srv1429066262:84 , srv1468122706:84 , srv1551435071:84 , srv1580047667:84 , srv1593055697:84 , srv1628313014:84 , srv1649651363:84 , srv1659600541:84 , srv17283885:84 , srv180098678:84 , srv1940483854:84 , srv2004856143:84 , srv204107055:84 , srv2114266643:84 , srv2137685854:84 , srv334937685:84 , srv367091719:84 , srv402376795:84 , srv416155699:84 , srv451592021:84 , srv455974126:84 , srv462398044:84 , srv463030025:84 , srv514447207:84 , srv642725057:84 , srv710666253:84 , srv766219667:84 , srv821888787:84 , srv863862794:84 , srv890260389:84 , srv913746800:84 , srv949105195:84 , srv952992063:84 , srv954659345:84 , srv962780576:84 , srv984860820:84 , srv1054283001:85 , srv1069793759:85 , srv1211473304:85 , srv1254464242:85 , srv1283249812:85 , srv1370568293:85 , srv1464112739:85 , srv1466630446:85 , srv160756848:85 , srv1636442060:85 , srv1642540118:85 , srv1647175964:85 , srv1666838767:85 , srv1794468048:85 , srv1838304457:85 , srv1845908376:85 , srv1848332710:85 , srv1999137180:85 , srv2028757447:85 , srv205692560:85 , srv2108442362:85 , srv240573768:85 , srv254381240:85 , srv295405867:85 , srv327537199:85 , srv44029806:85 , srv485187435:85 , srv528638802:85 , srv538337901:85 , srv546814221:85 , srv578025858:85 , srv579804826:85 , srv635600664:85 , srv805036295:85 , srv880813904:85 , srv1108745995:86 , srv132855945:86 , srv1460550654:86 , srv1734663898:86 , srv1764826202:86 , srv188565480:86 , srv667872149:86 , srv1313720909:643 }
2024-03-01 12:53:20,919 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=274 (was 274), ProcessCount=171 (was 171), AvailableMemoryMB=9852 (was 10609)
2024-03-01 12:53:20,932 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=274, ProcessCount=171, AvailableMemoryMB=9851
2024-03-01 12:53:20,933 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-01 12:53:20,933 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-01 12:53:20,942 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020032566:84 , srv1085704066:84 , srv1302999212:84 , srv1399840537:84 , srv1533882501:84 , srv1543268403:84 , srv1572835723:84 , srv1610603854:84 , srv1613145461:84 , srv173755256:84 , srv1774400186:84 , srv184092613:84 , srv1993410718:84 , srv2043928773:84 , srv2065056518:84 , srv257257859:84 , srv268458306:84 , srv38226121:84 , srv405571561:84 , srv451885286:84 , srv504065674:84 , srv530731315:84 , srv566624805:84 , srv592385815:84 , srv633696647:84 , srv662577544:84 , srv936417900:84 , srv951337380:84 , srv978942326:84 , srv1481811745:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-01 12:53:20,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:53:20,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1543268403=6, srv184092613=12, srv451885286=20, srv504065674=21, srv257257859=16, srv633696647=25, srv1613145461=9, srv978942326=29, srv936417900=27, srv1993410718=13, srv592385815=24, srv566624805=23, srv1020032566=0, srv1085704066=1, srv2043928773=14, srv1399840537=3, srv268458306=17, srv173755256=10, srv951337380=28, srv530731315=22, srv662577544=26, srv1533882501=5, srv1481811745=4, srv1302999212=2, srv405571561=19, srv1774400186=11, srv38226121=18, srv2065056518=15, srv1572835723=7, srv1610603854=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:53:20,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:53:20,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:53:20,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:53:20,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:53:20,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:53:20,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:53:20,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:53:20,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:53:20,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:53:20,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:53:20,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-01 12:53:20,968 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1753329301650655, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:53:20,968 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912745511602081, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1753329301650655, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-01 12:55:20,957 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6093696 different iterations.  Found a solution that moves 1861 regions; Going from a computed imbalance of 0.9912745511602081 to a new imbalance of 4.3436668390529226E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6892592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:55:20,963 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020032566:90 , srv1085704066:90 , srv1302999212:90 , srv1399840537:90 , srv1481811745:90 , srv1533882501:90 , srv1543268403:90 , srv1572835723:90 , srv1610603854:90 , srv1613145461:90 , srv173755256:90 , srv1774400186:90 , srv184092613:90 , srv1993410718:90 , srv2043928773:90 , srv2065056518:90 , srv257257859:90 , srv268458306:90 , srv38226121:90 , srv405571561:90 , srv451885286:90 , srv504065674:90 , srv530731315:90 , srv566624805:90 , srv592385815:90 , srv633696647:90 , srv662577544:90 , srv936417900:90 , srv951337380:90 , srv978942326:90 }
2024-03-01 12:55:20,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-01 12:55:20,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1543268403=6, srv184092613=12, srv451885286=20, srv504065674=21, srv257257859=16, srv633696647=25, srv1613145461=9, srv978942326=29, srv936417900=27, srv1993410718=13, srv592385815=24, srv566624805=23, srv1020032566=0, srv1085704066=1, srv2043928773=14, srv1399840537=3, srv268458306=17, srv173755256=10, srv951337380=28, srv530731315=22, srv662577544=26, srv1533882501=5, srv1481811745=4, srv1302999212=2, srv405571561=19, srv1774400186=11, srv38226121=18, srv2065056518=15, srv1572835723=7, srv1610603854=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-01 12:55:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-01 12:55:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-01 12:55:20,991 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-01 12:55:20,992 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1020032566:90 , srv1085704066:90 , srv1302999212:90 , srv1399840537:90 , srv1481811745:90 , srv1533882501:90 , srv1543268403:90 , srv1572835723:90 , srv1610603854:90 , srv1613145461:90 , srv173755256:90 , srv1774400186:90 , srv184092613:90 , srv1993410718:90 , srv2043928773:90 , srv2065056518:90 , srv257257859:90 , srv268458306:90 , srv38226121:90 , srv405571561:90 , srv451885286:90 , srv504065674:90 , srv530731315:90 , srv566624805:90 , srv592385815:90 , srv633696647:90 , srv662577544:90 , srv936417900:90 , srv951337380:90 , srv978942326:90 }
2024-03-01 12:55:21,008 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 274) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=8988 (was 9851)