2023-08-05 08:53:10,771 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-08-05 08:53:11,105 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-08-05 08:53:11,218 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-08-05 08:53:11,287 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-05 08:53:11,296 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-05 08:53:11,311 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=273, ProcessCount=173, AvailableMemoryMB=2433
2023-08-05 08:53:11,313 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-05 08:53:11,314 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-05 08:53:11,472 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009072710:84 , srv1035293113:84 , srv1045223958:84 , srv1086011406:84 , srv1134742730:84 , srv117123625:84 , srv1178130203:84 , srv130593033:84 , srv1310891581:84 , srv1312862093:84 , srv1345921084:84 , srv1372445006:84 , srv1375625643:84 , srv1378004278:84 , srv1393698835:84 , srv1397939510:84 , srv1488757237:84 , srv1499098574:84 , srv1501374367:84 , srv1518237074:84 , srv1533813436:84 , srv1607522825:84 , srv1622209918:84 , srv1641448043:84 , srv1645842166:84 , srv1661395366:84 , srv1663340826:84 , srv1668184081:84 , srv1670356540:84 , srv167378223:84 , srv1735777762:84 , srv1758081959:84 , srv1760369632:84 , srv1777823123:84 , srv1813053913:84 , srv1849460809:84 , srv1886539257:84 , srv1893536076:84 , srv189985905:84 , srv1901003284:84 , srv1943007239:84 , srv1948881778:84 , srv1960043989:84 , srv1972791861:84 , srv1976846943:84 , srv1986860438:84 , srv1993736609:84 , srv1995445253:84 , srv200891838:84 , srv2013029029:84 , srv2092134050:84 , srv2092983940:84 , srv2095890754:84 , srv2116344541:84 , srv2119969492:84 , srv245679878:84 , srv281736433:84 , srv289726240:84 , srv353725217:84 , srv366058720:84 , srv367016532:84 , srv374148604:84 , srv382794248:84 , srv390584506:84 , srv427797027:84 , srv428570418:84 , srv434050261:84 , srv443391761:84 , srv458946296:84 , srv477731673:84 , srv487741365:84 , srv524964710:84 , srv546322624:84 , srv556714473:84 , srv667358635:84 , srv674906022:84 , srv683296545:84 , srv748235503:84 , srv755606012:84 , srv756424717:84 , srv773172055:84 , srv780197212:84 , srv787547876:84 , srv832321749:84 , srv851336218:84 , srv851687147:84 , srv859142566:84 , srv865508353:84 , srv876591357:84 , srv883912483:84 , srv904806230:84 , srv920722067:84 , srv924354104:84 , srv940744256:84 , srv941758898:84 , srv960899439:84 , srv964883993:84 , srv968563868:84 , srv998034661:84 , srv586371103:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-08-05 08:53:11,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:53:11,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2116344541=53, srv924354104=93, srv780197212=82, srv883912483=90, srv851336218=85, srv1533813436=20, srv1735777762=30, srv1670356540=28, srv851687147=86, srv2119969492=54, srv2092134050=50, srv1378004278=13, srv189985905=38, srv787547876=83, srv920722067=92, srv904806230=91, srv1518237074=19, srv1777823123=33, srv1045223958=2, srv367016532=60, srv281736433=56, srv1668184081=27, srv859142566=87, srv865508353=88, srv1372445006=11, srv353725217=58, srv998034661=99, srv556714473=73, srv1134742730=4, srv1375625643=12, srv390584506=63, srv1312862093=9, srv546322624=72, srv1976846943=44, srv674906022=76, srv1986860438=45, srv1760369632=32, srv1886539257=36, srv200891838=48, srv1488757237=16, srv443391761=67, srv832321749=84, srv477731673=69, srv1893536076=37, srv434050261=66, srv458946296=68, srv1641448043=23, srv487741365=70, srv968563868=98, srv1086011406=3, srv1499098574=17, srv1009072710=0, srv1849460809=35, srv586371103=74, srv374148604=61, srv382794248=62, srv1501374367=18, srv755606012=79, srv167378223=29, srv427797027=64, srv748235503=78, srv1607522825=21, srv667358635=75, srv130593033=7, srv1645842166=24, srv1397939510=15, srv1345921084=10, srv756424717=80, srv1758081959=31, srv1035293113=1, srv1901003284=39, srv773172055=81, srv960899439=96, srv1661395366=25, srv1663340826=26, srv1960043989=42, srv941758898=95, srv876591357=89, srv683296545=77, srv366058720=59, srv1622209918=22, srv1813053913=34, srv1178130203=6, srv2092983940=51, srv245679878=55, srv117123625=5, srv1995445253=47, srv964883993=97, srv2095890754=52, srv524964710=71, srv1972791861=43, srv1948881778=41, srv940744256=94, srv1393698835=14, srv1993736609=46, srv1943007239=40, srv1310891581=8, srv289726240=57, srv428570418=65, srv2013029029=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:53:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:53:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:53:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:53:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:53:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:53:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:53:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:53:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:53:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:53:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:53:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:53:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:53:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:53:11,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-05 08:53:11,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-05 08:53:11,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-05 08:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-05 08:53:11,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-05 08:53:11,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-05 08:53:11,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-05 08:53:11,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-05 08:53:11,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-05 08:53:11,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-05 08:53:11,752 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:11,752 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-05 08:53:11,753 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-05 08:53:12,092 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 490 ms to try 5000 different iterations.  Found a solution that moves 4677 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18280767125342884. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03760838765220106, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050605758179704814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5196666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03760838765220106, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:12,115 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1397939510:82 , srv1533813436:82 , srv1607522825:82 , srv1758081959:82 , srv1972791861:82 , srv443391761:82 , srv748235503:82 , srv773172055:82 , srv920722067:82 , srv2116344541:83 , srv281736433:83 , srv477731673:83 , srv1086011406:84 , srv1518237074:84 , srv1995445253:84 , srv289726240:84 , srv458946296:84 , srv667358635:84 , srv851687147:84 , srv904806230:84 , srv1009072710:85 , srv1312862093:85 , srv1501374367:85 , srv1668184081:85 , srv189985905:85 , srv1986860438:85 , srv2119969492:85 , srv366058720:85 , srv382794248:85 , srv546322624:85 , srv683296545:85 , srv787547876:85 , srv859142566:85 , srv876591357:85 , srv964883993:85 , srv1035293113:86 , srv1134742730:86 , srv1345921084:86 , srv1372445006:86 , srv1499098574:86 , srv167378223:86 , srv1976846943:86 , srv1993736609:86 , srv245679878:86 , srv374148604:86 , srv434050261:86 , srv524964710:86 , srv780197212:86 , srv924354104:86 , srv130593033:87 , srv1310891581:87 , srv1378004278:87 , srv1641448043:87 , srv1893536076:87 , srv1943007239:87 , srv1948881778:87 , srv2095890754:87 , srv390584506:87 , srv428570418:87 , srv487741365:87 , srv756424717:87 , srv832321749:87 , srv851336218:87 , srv117123625:88 , srv1645842166:88 , srv1661395366:88 , srv1670356540:88 , srv1760369632:88 , srv1849460809:88 , srv1886539257:88 , srv1901003284:88 , srv1960043989:88 , srv353725217:88 , srv556714473:88 , srv755606012:88 , srv865508353:88 , srv883912483:88 , srv941758898:88 , srv960899439:88 , srv1375625643:89 , srv1393698835:89 , srv1663340826:89 , srv1813053913:89 , srv2092983940:89 , srv968563868:89 , srv998034661:89 , srv1178130203:90 , srv1488757237:90 , srv367016532:90 , srv674906022:90 , srv1622209918:91 , srv2013029029:91 , srv427797027:91 , srv940744256:91 , srv1045223958:92 , srv1777823123:92 , srv200891838:92 , srv1735777762:94 , srv2092134050:95 , srv586371103:424 }
2023-08-05 08:53:12,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:53:12,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2116344541=53, srv924354104=93, srv780197212=82, srv883912483=90, srv851336218=85, srv1533813436=20, srv1735777762=30, srv1670356540=28, srv851687147=86, srv2119969492=54, srv2092134050=50, srv1378004278=13, srv189985905=38, srv787547876=83, srv920722067=92, srv904806230=91, srv1518237074=19, srv1777823123=33, srv1045223958=2, srv367016532=60, srv281736433=56, srv1668184081=27, srv859142566=87, srv865508353=88, srv1372445006=11, srv353725217=58, srv998034661=99, srv556714473=73, srv1134742730=4, srv1375625643=12, srv390584506=63, srv1312862093=9, srv546322624=72, srv1976846943=44, srv674906022=76, srv1986860438=45, srv1760369632=32, srv1886539257=36, srv200891838=48, srv1488757237=16, srv443391761=67, srv832321749=84, srv477731673=69, srv1893536076=37, srv434050261=66, srv458946296=68, srv1641448043=23, srv487741365=70, srv968563868=98, srv1086011406=3, srv1499098574=17, srv1009072710=0, srv1849460809=35, srv586371103=74, srv374148604=61, srv382794248=62, srv1501374367=18, srv755606012=79, srv167378223=29, srv427797027=64, srv748235503=78, srv1607522825=21, srv667358635=75, srv130593033=7, srv1645842166=24, srv1397939510=15, srv1345921084=10, srv756424717=80, srv1758081959=31, srv1035293113=1, srv1901003284=39, srv773172055=81, srv960899439=96, srv1661395366=25, srv1663340826=26, srv1960043989=42, srv941758898=95, srv876591357=89, srv683296545=77, srv366058720=59, srv1622209918=22, srv1813053913=34, srv1178130203=6, srv2092983940=51, srv245679878=55, srv117123625=5, srv1995445253=47, srv964883993=97, srv2095890754=52, srv524964710=71, srv1972791861=43, srv1948881778=41, srv940744256=94, srv1393698835=14, srv1993736609=46, srv1943007239=40, srv1310891581=8, srv289726240=57, srv428570418=65, srv2013029029=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:53:12,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:53:12,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:53:12,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-05 08:53:12,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-05 08:53:12,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-05 08:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-05 08:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-05 08:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-05 08:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-05 08:53:12,268 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03760838765220106, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050605758179704814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03760838765220106, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:12,268 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-05 08:53:12,269 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18277492219946928, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03760838765220106, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050605758179704814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03760838765220106, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-05 08:53:12,576 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 423 ms to try 5000 different iterations.  Found a solution that moves 2670 regions; Going from a computed imbalance of 0.18277492219946928 to a new imbalance of 0.012970086150548227. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020061011814880812); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03240456938030165, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2966666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020061011814880812); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:12,586 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1735777762:76 , srv2092134050:77 , srv200891838:79 , srv1045223958:80 , srv1375625643:80 , srv2013029029:80 , srv940744256:80 , srv427797027:81 , srv968563868:81 , srv998034661:81 , srv1393698835:82 , srv1488757237:82 , srv1622209918:82 , srv1663340826:82 , srv1670356540:82 , srv941758898:82 , srv960899439:82 , srv1777823123:83 , srv1901003284:83 , srv556714473:83 , srv674906022:83 , srv883912483:83 , srv130593033:84 , srv1345921084:84 , srv1641448043:84 , srv1645842166:84 , srv167378223:84 , srv1813053913:84 , srv1886539257:84 , srv1948881778:84 , srv2095890754:84 , srv353725217:84 , srv367016532:84 , srv1035293113:85 , srv117123625:85 , srv1849460809:85 , srv1893536076:85 , srv1943007239:85 , srv487741365:85 , srv524964710:85 , srv787547876:85 , srv859142566:85 , srv865508353:85 , srv1134742730:86 , srv1178130203:86 , srv1310891581:86 , srv1661395366:86 , srv1668184081:86 , srv1760369632:86 , srv1960043989:86 , srv1976846943:86 , srv1986860438:86 , srv1993736609:86 , srv2092983940:86 , srv2119969492:86 , srv245679878:86 , srv374148604:86 , srv390584506:86 , srv546322624:86 , srv756424717:86 , srv832321749:86 , srv1086011406:87 , srv189985905:87 , srv428570418:87 , srv667358635:87 , srv851336218:87 , srv851687147:87 , srv876591357:87 , srv964883993:87 , srv1372445006:88 , srv1378004278:88 , srv1397939510:88 , srv1499098574:88 , srv1501374367:88 , srv1518237074:88 , srv1995445253:88 , srv2116344541:88 , srv289726240:88 , srv366058720:88 , srv382794248:88 , srv443391761:88 , srv458946296:88 , srv755606012:88 , srv924354104:88 , srv1009072710:89 , srv1312862093:89 , srv1533813436:89 , srv1972791861:89 , srv281736433:89 , srv683296545:89 , srv748235503:89 , srv434050261:90 , srv773172055:90 , srv904806230:90 , srv920722067:90 , srv1758081959:91 , srv1607522825:92 , srv477731673:92 , srv780197212:92 , srv586371103:528 }
2023-08-05 08:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2116344541=53, srv924354104=93, srv780197212=82, srv883912483=90, srv851336218=85, srv1533813436=20, srv1735777762=30, srv1670356540=28, srv851687147=86, srv2119969492=54, srv2092134050=50, srv1378004278=13, srv189985905=38, srv787547876=83, srv920722067=92, srv904806230=91, srv1518237074=19, srv1777823123=33, srv1045223958=2, srv367016532=60, srv281736433=56, srv1668184081=27, srv859142566=87, srv865508353=88, srv1372445006=11, srv353725217=58, srv998034661=99, srv556714473=73, srv1134742730=4, srv1375625643=12, srv390584506=63, srv1312862093=9, srv546322624=72, srv1976846943=44, srv674906022=76, srv1986860438=45, srv1760369632=32, srv1886539257=36, srv200891838=48, srv1488757237=16, srv443391761=67, srv832321749=84, srv477731673=69, srv1893536076=37, srv434050261=66, srv458946296=68, srv1641448043=23, srv487741365=70, srv968563868=98, srv1086011406=3, srv1499098574=17, srv1009072710=0, srv1849460809=35, srv586371103=74, srv374148604=61, srv382794248=62, srv1501374367=18, srv755606012=79, srv167378223=29, srv427797027=64, srv748235503=78, srv1607522825=21, srv667358635=75, srv130593033=7, srv1645842166=24, srv1397939510=15, srv1345921084=10, srv756424717=80, srv1758081959=31, srv1035293113=1, srv1901003284=39, srv773172055=81, srv960899439=96, srv1661395366=25, srv1663340826=26, srv1960043989=42, srv941758898=95, srv876591357=89, srv683296545=77, srv366058720=59, srv1622209918=22, srv1813053913=34, srv1178130203=6, srv2092983940=51, srv245679878=55, srv117123625=5, srv1995445253=47, srv964883993=97, srv2095890754=52, srv524964710=71, srv1972791861=43, srv1948881778=41, srv940744256=94, srv1393698835=14, srv1993736609=46, srv1943007239=40, srv1310891581=8, srv289726240=57, srv428570418=65, srv2013029029=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-05 08:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-05 08:53:12,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-05 08:53:12,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-05 08:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-05 08:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-05 08:53:12,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-05 08:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-05 08:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-05 08:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-05 08:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-05 08:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-05 08:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-05 08:53:12,795 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020061011814880812); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03240456938030165, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020061011814880812); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:12,795 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-05 08:53:12,796 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012951390410956174, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020061011814880812); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03240456938030165, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020061011814880812); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-05 08:53:13,129 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 479 ms to try 5000 different iterations.  Found a solution that moves 969 regions; Going from a computed imbalance of 0.012951390410956174 to a new imbalance of 3.865110250068937E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006659180233461053); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005565647399578146); MoveCostFunction : (multiplier=7.0, imbalance=0.10766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006659180233461053); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:13,131 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv780197212:80 , srv477731673:81 , srv755606012:81 , srv1178130203:82 , srv1661395366:82 , srv1777823123:82 , srv2092983940:82 , srv434050261:82 , srv117123625:83 , srv1372445006:83 , srv1378004278:83 , srv1499098574:83 , srv1607522825:83 , srv1760369632:83 , srv1960043989:83 , srv924354104:83 , srv1009072710:84 , srv1312862093:84 , srv1501374367:84 , srv1622209918:84 , srv367016532:84 , srv382794248:84 , srv390584506:84 , srv428570418:84 , srv674906022:84 , srv683296545:84 , srv851336218:84 , srv865508353:84 , srv904806230:84 , srv1045223958:85 , srv1134742730:85 , srv1310891581:85 , srv1488757237:85 , srv1758081959:85 , srv1813053913:85 , srv1849460809:85 , srv1886539257:85 , srv189985905:85 , srv1976846943:85 , srv1986860438:85 , srv1993736609:85 , srv2092134050:85 , srv245679878:85 , srv281736433:85 , srv289726240:85 , srv353725217:85 , srv427797027:85 , srv487741365:85 , srv756424717:85 , srv773172055:85 , srv832321749:85 , srv876591357:85 , srv920722067:85 , srv964883993:85 , srv1035293113:86 , srv1086011406:86 , srv1393698835:86 , srv1518237074:86 , srv1533813436:86 , srv1645842166:86 , srv1663340826:86 , srv1943007239:86 , srv1948881778:86 , srv1972791861:86 , srv1995445253:86 , srv2013029029:86 , srv2095890754:86 , srv2116344541:86 , srv2119969492:86 , srv366058720:86 , srv374148604:86 , srv458946296:86 , srv667358635:86 , srv748235503:86 , srv851687147:86 , srv883912483:86 , srv130593033:87 , srv1397939510:87 , srv1668184081:87 , srv167378223:87 , srv1893536076:87 , srv1901003284:87 , srv200891838:87 , srv443391761:87 , srv524964710:87 , srv546322624:87 , srv556714473:87 , srv859142566:87 , srv940744256:87 , srv941758898:87 , srv998034661:87 , srv1345921084:88 , srv1375625643:88 , srv1641448043:88 , srv1670356540:88 , srv1735777762:88 , srv787547876:88 , srv968563868:88 , srv960899439:89 , srv586371103:565 }
2023-08-05 08:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2116344541=53, srv924354104=93, srv780197212=82, srv883912483=90, srv851336218=85, srv1533813436=20, srv1735777762=30, srv1670356540=28, srv851687147=86, srv2119969492=54, srv2092134050=50, srv1378004278=13, srv189985905=38, srv787547876=83, srv920722067=92, srv904806230=91, srv1518237074=19, srv1777823123=33, srv1045223958=2, srv367016532=60, srv281736433=56, srv1668184081=27, srv859142566=87, srv865508353=88, srv1372445006=11, srv353725217=58, srv998034661=99, srv556714473=73, srv1134742730=4, srv1375625643=12, srv390584506=63, srv1312862093=9, srv546322624=72, srv1976846943=44, srv674906022=76, srv1986860438=45, srv1760369632=32, srv1886539257=36, srv200891838=48, srv1488757237=16, srv443391761=67, srv832321749=84, srv477731673=69, srv1893536076=37, srv434050261=66, srv458946296=68, srv1641448043=23, srv487741365=70, srv968563868=98, srv1086011406=3, srv1499098574=17, srv1009072710=0, srv1849460809=35, srv586371103=74, srv374148604=61, srv382794248=62, srv1501374367=18, srv755606012=79, srv167378223=29, srv427797027=64, srv748235503=78, srv1607522825=21, srv667358635=75, srv130593033=7, srv1645842166=24, srv1397939510=15, srv1345921084=10, srv756424717=80, srv1758081959=31, srv1035293113=1, srv1901003284=39, srv773172055=81, srv960899439=96, srv1661395366=25, srv1663340826=26, srv1960043989=42, srv941758898=95, srv876591357=89, srv683296545=77, srv366058720=59, srv1622209918=22, srv1813053913=34, srv1178130203=6, srv2092983940=51, srv245679878=55, srv117123625=5, srv1995445253=47, srv964883993=97, srv2095890754=52, srv524964710=71, srv1972791861=43, srv1948881778=41, srv940744256=94, srv1393698835=14, srv1993736609=46, srv1943007239=40, srv1310891581=8, srv289726240=57, srv428570418=65, srv2013029029=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:53:13,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-05 08:53:13,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-05 08:53:13,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-05 08:53:13,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-05 08:53:13,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-05 08:53:13,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-05 08:53:13,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-05 08:53:13,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-05 08:53:13,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-05 08:53:13,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-05 08:53:13,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006659180233461053); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005565647399578146); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006659180233461053); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:13,289 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-05 08:53:13,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.7972594198640644E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006659180233461053); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005565647399578146); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006659180233461053); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-05 08:53:13,719 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 543 ms to try 5000 different iterations.  Found a solution that moves 196 regions; Going from a computed imbalance of 3.7972594198640644E-4 to a new imbalance of 1.8576635508475874E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.021777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:13,720 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1893536076:83 , srv366058720:83 , srv851336218:83 , srv960899439:83 , srv1009072710:84 , srv1178130203:84 , srv130593033:84 , srv1310891581:84 , srv1312862093:84 , srv1345921084:84 , srv1378004278:84 , srv1518237074:84 , srv1607522825:84 , srv1622209918:84 , srv1641448043:84 , srv1645842166:84 , srv1668184081:84 , srv1670356540:84 , srv1735777762:84 , srv1758081959:84 , srv1777823123:84 , srv1813053913:84 , srv1849460809:84 , srv1901003284:84 , srv1943007239:84 , srv1976846943:84 , srv1993736609:84 , srv1995445253:84 , srv200891838:84 , srv374148604:84 , srv434050261:84 , srv458946296:84 , srv524964710:84 , srv546322624:84 , srv556714473:84 , srv683296545:84 , srv756424717:84 , srv780197212:84 , srv832321749:84 , srv904806230:84 , srv940744256:84 , srv968563868:84 , srv1035293113:85 , srv1045223958:85 , srv1086011406:85 , srv1134742730:85 , srv1372445006:85 , srv1375625643:85 , srv1393698835:85 , srv1397939510:85 , srv1488757237:85 , srv1499098574:85 , srv1501374367:85 , srv1533813436:85 , srv1663340826:85 , srv167378223:85 , srv1760369632:85 , srv1886539257:85 , srv1948881778:85 , srv1960043989:85 , srv1972791861:85 , srv2013029029:85 , srv2092134050:85 , srv2092983940:85 , srv2095890754:85 , srv2116344541:85 , srv2119969492:85 , srv281736433:85 , srv289726240:85 , srv353725217:85 , srv367016532:85 , srv382794248:85 , srv390584506:85 , srv427797027:85 , srv428570418:85 , srv443391761:85 , srv477731673:85 , srv487741365:85 , srv667358635:85 , srv674906022:85 , srv748235503:85 , srv755606012:85 , srv773172055:85 , srv787547876:85 , srv851687147:85 , srv859142566:85 , srv865508353:85 , srv883912483:85 , srv920722067:85 , srv924354104:85 , srv941758898:85 , srv964883993:85 , srv998034661:85 , srv117123625:86 , srv1661395366:86 , srv189985905:86 , srv1986860438:86 , srv245679878:86 , srv876591357:86 , srv586371103:625 }
2023-08-05 08:53:13,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:53:13,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2116344541=53, srv924354104=93, srv780197212=82, srv883912483=90, srv851336218=85, srv1533813436=20, srv1735777762=30, srv1670356540=28, srv851687147=86, srv2119969492=54, srv2092134050=50, srv1378004278=13, srv189985905=38, srv787547876=83, srv920722067=92, srv904806230=91, srv1518237074=19, srv1777823123=33, srv1045223958=2, srv367016532=60, srv281736433=56, srv1668184081=27, srv859142566=87, srv865508353=88, srv1372445006=11, srv353725217=58, srv998034661=99, srv556714473=73, srv1134742730=4, srv1375625643=12, srv390584506=63, srv1312862093=9, srv546322624=72, srv1976846943=44, srv674906022=76, srv1986860438=45, srv1760369632=32, srv1886539257=36, srv200891838=48, srv1488757237=16, srv443391761=67, srv832321749=84, srv477731673=69, srv1893536076=37, srv434050261=66, srv458946296=68, srv1641448043=23, srv487741365=70, srv968563868=98, srv1086011406=3, srv1499098574=17, srv1009072710=0, srv1849460809=35, srv586371103=74, srv374148604=61, srv382794248=62, srv1501374367=18, srv755606012=79, srv167378223=29, srv427797027=64, srv748235503=78, srv1607522825=21, srv667358635=75, srv130593033=7, srv1645842166=24, srv1397939510=15, srv1345921084=10, srv756424717=80, srv1758081959=31, srv1035293113=1, srv1901003284=39, srv773172055=81, srv960899439=96, srv1661395366=25, srv1663340826=26, srv1960043989=42, srv941758898=95, srv876591357=89, srv683296545=77, srv366058720=59, srv1622209918=22, srv1813053913=34, srv1178130203=6, srv2092983940=51, srv245679878=55, srv117123625=5, srv1995445253=47, srv964883993=97, srv2095890754=52, srv524964710=71, srv1972791861=43, srv1948881778=41, srv940744256=94, srv1393698835=14, srv1993736609=46, srv1943007239=40, srv1310891581=8, srv289726240=57, srv428570418=65, srv2013029029=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:53:13,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-05 08:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-05 08:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-05 08:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-05 08:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-05 08:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-05 08:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-05 08:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-05 08:53:13,912 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7204214175126534E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:13,912 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1893536076:83 , srv366058720:83 , srv851336218:83 , srv960899439:83 , srv1009072710:84 , srv1178130203:84 , srv130593033:84 , srv1310891581:84 , srv1312862093:84 , srv1345921084:84 , srv1378004278:84 , srv1518237074:84 , srv1607522825:84 , srv1622209918:84 , srv1641448043:84 , srv1645842166:84 , srv1668184081:84 , srv1670356540:84 , srv1735777762:84 , srv1758081959:84 , srv1777823123:84 , srv1813053913:84 , srv1849460809:84 , srv1901003284:84 , srv1943007239:84 , srv1976846943:84 , srv1993736609:84 , srv1995445253:84 , srv200891838:84 , srv374148604:84 , srv434050261:84 , srv458946296:84 , srv524964710:84 , srv546322624:84 , srv556714473:84 , srv683296545:84 , srv756424717:84 , srv780197212:84 , srv832321749:84 , srv904806230:84 , srv940744256:84 , srv968563868:84 , srv1035293113:85 , srv1045223958:85 , srv1086011406:85 , srv1134742730:85 , srv1372445006:85 , srv1375625643:85 , srv1393698835:85 , srv1397939510:85 , srv1488757237:85 , srv1499098574:85 , srv1501374367:85 , srv1533813436:85 , srv1663340826:85 , srv167378223:85 , srv1760369632:85 , srv1886539257:85 , srv1948881778:85 , srv1960043989:85 , srv1972791861:85 , srv2013029029:85 , srv2092134050:85 , srv2092983940:85 , srv2095890754:85 , srv2116344541:85 , srv2119969492:85 , srv281736433:85 , srv289726240:85 , srv353725217:85 , srv367016532:85 , srv382794248:85 , srv390584506:85 , srv427797027:85 , srv428570418:85 , srv443391761:85 , srv477731673:85 , srv487741365:85 , srv667358635:85 , srv674906022:85 , srv748235503:85 , srv755606012:85 , srv773172055:85 , srv787547876:85 , srv851687147:85 , srv859142566:85 , srv865508353:85 , srv883912483:85 , srv920722067:85 , srv924354104:85 , srv941758898:85 , srv964883993:85 , srv998034661:85 , srv117123625:86 , srv1661395366:86 , srv189985905:86 , srv1986860438:86 , srv245679878:86 , srv876591357:86 , srv586371103:625 }
2023-08-05 08:53:13,951 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=273 (was 273), ProcessCount=173 (was 173), AvailableMemoryMB=1736 (was 2433)
2023-08-05 08:53:13,963 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=273, ProcessCount=173, AvailableMemoryMB=1735
2023-08-05 08:53:13,964 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-05 08:53:13,964 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-08-05 08:53:13,981 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005330610:84 , srv1058131802:84 , srv1117609887:84 , srv114521170:84 , srv1152233446:84 , srv1216983076:84 , srv1262552216:84 , srv1433808858:84 , srv146503394:84 , srv1479717613:84 , srv1647593640:84 , srv1658009719:84 , srv1684919212:84 , srv2126388683:84 , srv216365123:84 , srv241892384:84 , srv278261536:84 , srv28057224:84 , srv394584405:84 , srv405504307:84 , srv485529840:84 , srv510082914:84 , srv553339360:84 , srv635011046:84 , srv658762317:84 , srv87239726:84 , srv923378698:84 , srv962503342:84 , srv97271699:84 , srv1080681463:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-08-05 08:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1152233446=5, srv635011046=24, srv1658009719=12, srv1058131802=1, srv962503342=28, srv1433808858=8, srv2126388683=14, srv485529840=21, srv923378698=27, srv97271699=29, srv658762317=25, srv1647593640=11, srv1684919212=13, srv216365123=15, srv28057224=18, srv1262552216=7, srv146503394=9, srv241892384=16, srv553339360=23, srv87239726=26, srv1117609887=3, srv510082914=22, srv114521170=4, srv1216983076=6, srv1005330610=0, srv1080681463=2, srv1479717613=10, srv405504307=20, srv394584405=19, srv278261536=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:53:14,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:53:14,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:53:14,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:53:14,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:53:14,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:53:14,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:53:14,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:53:14,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:53:14,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:53:14,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-05 08:53:14,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1140585093214583, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:53:14,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.99125524379028, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1140585093214583, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-08-05 08:55:14,011 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5790101 different iterations.  Found a solution that moves 1863 regions; Going from a computed imbalance of 0.99125524379028 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:55:14,017 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005330610:90 , srv1058131802:90 , srv1080681463:90 , srv1117609887:90 , srv114521170:90 , srv1152233446:90 , srv1216983076:90 , srv1262552216:90 , srv1433808858:90 , srv146503394:90 , srv1479717613:90 , srv1647593640:90 , srv1658009719:90 , srv1684919212:90 , srv2126388683:90 , srv216365123:90 , srv241892384:90 , srv278261536:90 , srv28057224:90 , srv394584405:90 , srv405504307:90 , srv485529840:90 , srv510082914:90 , srv553339360:90 , srv635011046:90 , srv658762317:90 , srv87239726:90 , srv923378698:90 , srv962503342:90 , srv97271699:90 }
2023-08-05 08:55:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-05 08:55:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1152233446=5, srv635011046=24, srv1658009719=12, srv1058131802=1, srv962503342=28, srv1433808858=8, srv2126388683=14, srv485529840=21, srv923378698=27, srv97271699=29, srv658762317=25, srv1647593640=11, srv1684919212=13, srv216365123=15, srv28057224=18, srv1262552216=7, srv146503394=9, srv241892384=16, srv553339360=23, srv87239726=26, srv1117609887=3, srv510082914=22, srv114521170=4, srv1216983076=6, srv1005330610=0, srv1080681463=2, srv1479717613=10, srv405504307=20, srv394584405=19, srv278261536=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-05 08:55:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-05 08:55:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-05 08:55:14,037 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-05 08:55:14,038 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1005330610:90 , srv1058131802:90 , srv1080681463:90 , srv1117609887:90 , srv114521170:90 , srv1152233446:90 , srv1216983076:90 , srv1262552216:90 , srv1433808858:90 , srv146503394:90 , srv1479717613:90 , srv1647593640:90 , srv1658009719:90 , srv1684919212:90 , srv2126388683:90 , srv216365123:90 , srv241892384:90 , srv278261536:90 , srv28057224:90 , srv394584405:90 , srv405504307:90 , srv485529840:90 , srv510082914:90 , srv553339360:90 , srv635011046:90 , srv658762317:90 , srv87239726:90 , srv923378698:90 , srv962503342:90 , srv97271699:90 }
2023-08-05 08:55:14,057 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=146 (was 273), ProcessCount=171 (was 173), AvailableMemoryMB=1738 (was 1735) - AvailableMemoryMB LEAK? -