2023-09-16 02:53:13,783 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-16 02:53:14,108 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-16 02:53:14,215 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-16 02:53:14,294 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-16 02:53:14,301 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-16 02:53:14,315 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=169, AvailableMemoryMB=7062
2023-09-16 02:53:14,318 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-16 02:53:14,318 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-16 02:53:14,450 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003006021:84 , srv1029454692:84 , srv1034205543:84 , srv1041925141:84 , srv1050525702:84 , srv1085134172:84 , srv1117254217:84 , srv1118592712:84 , srv1147619936:84 , srv1160227009:84 , srv1175682344:84 , srv1211016829:84 , srv1234383439:84 , srv1241859614:84 , srv1268972875:84 , srv1291447420:84 , srv1305139784:84 , srv1359498632:84 , srv1364556464:84 , srv1377058761:84 , srv1385348286:84 , srv1398559989:84 , srv1418568030:84 , srv1419961700:84 , srv1429687180:84 , srv1484492778:84 , srv1490780931:84 , srv149656646:84 , srv1564795762:84 , srv1581141289:84 , srv1631511129:84 , srv1643707682:84 , srv167924700:84 , srv1684671021:84 , srv1685450761:84 , srv169351458:84 , srv1694311817:84 , srv1708320747:84 , srv1712628003:84 , srv1735214188:84 , srv1785109134:84 , srv1786029703:84 , srv178873696:84 , srv1791145707:84 , srv1829361696:84 , srv1829543476:84 , srv1837764886:84 , srv1871181948:84 , srv1877048959:84 , srv1909515908:84 , srv1929271820:84 , srv1932638794:84 , srv2000257019:84 , srv2003481804:84 , srv2115117753:84 , srv229600832:84 , srv249315381:84 , srv250650710:84 , srv257575640:84 , srv265169431:84 , srv297260296:84 , srv304031640:84 , srv327828967:84 , srv329236710:84 , srv344434943:84 , srv363145627:84 , srv37490782:84 , srv445363866:84 , srv463687759:84 , srv480324940:84 , srv480645640:84 , srv497297057:84 , srv49887054:84 , srv602936375:84 , srv604391702:84 , srv605265885:84 , srv639749905:84 , srv645616016:84 , srv646814768:84 , srv647188461:84 , srv661650747:84 , srv702525181:84 , srv703551040:84 , srv703826232:84 , srv729141042:84 , srv729590071:84 , srv743562353:84 , srv758456666:84 , srv783204527:84 , srv807145126:84 , srv834155604:84 , srv870048486:84 , srv876835412:84 , srv890034065:84 , srv892172798:84 , srv916118954:84 , srv98033623:84 , srv981509340:84 , srv995454128:84 , srv1055264409:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-16 02:53:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:53:14,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837764886=47, srv661650747=81, srv480324940=70, srv1160227009=10, srv1484492778=26, srv1490780931=27, srv1385348286=21, srv1685450761=35, srv344434943=65, srv329236710=64, srv604391702=75, srv1643707682=32, srv1684671021=34, srv1175682344=11, srv890034065=94, srv1305139784=17, srv1211016829=12, srv647188461=80, srv1055264409=5, srv1359498632=18, srv497297057=72, srv2115117753=55, srv1631511129=31, srv1029454692=1, srv645616016=78, srv257575640=59, srv1712628003=39, srv445363866=68, srv297260296=61, srv703826232=84, srv49887054=73, srv265169431=60, srv743562353=87, srv1268972875=15, srv639749905=77, srv602936375=74, srv480645640=71, srv169351458=36, srv1377058761=20, srv363145627=66, srv98033623=97, srv463687759=69, srv1085134172=6, srv1291447420=16, srv178873696=43, srv1829543476=46, srv646814768=79, srv892172798=95, srv995454128=99, srv1117254217=7, srv1929271820=51, srv1003006021=0, srv1909515908=50, srv1708320747=38, srv1581141289=30, srv1932638794=52, srv1429687180=25, srv1034205543=2, srv149656646=28, srv1241859614=14, srv1118592712=8, srv729141042=85, srv1791145707=44, srv2003481804=54, srv1050525702=4, srv1564795762=29, srv1871181948=48, srv2000257019=53, srv981509340=98, srv1694311817=37, srv1877048959=49, srv229600832=56, srv605265885=76, srv729590071=86, srv1786029703=42, srv1234383439=13, srv1419961700=24, srv807145126=90, srv250650710=58, srv916118954=96, srv870048486=92, srv167924700=33, srv1785109134=41, srv327828967=63, srv876835412=93, srv37490782=67, srv703551040=83, srv1418568030=23, srv1147619936=9, srv758456666=88, srv304031640=62, srv1398559989=22, srv834155604=91, srv1041925141=3, srv1829361696=45, srv1364556464=19, srv702525181=82, srv249315381=57, srv1735214188=40, srv783204527=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-16 02:53:14,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-16 02:53:14,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-16 02:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-16 02:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-16 02:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-16 02:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-16 02:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-16 02:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-16 02:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-16 02:53:14,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:14,718 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-16 02:53:14,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-16 02:53:15,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 528 ms to try 5000 different iterations.  Found a solution that moves 4730 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17368577304834812. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.043778398760458095, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05912189466459749, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5255555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.043778398760458095, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:15,121 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1234383439:80 , srv1871181948:80 , srv1241859614:81 , srv1291447420:81 , srv497297057:81 , srv1377058761:82 , srv1786029703:82 , srv178873696:82 , srv1791145707:82 , srv304031640:82 , srv480645640:82 , srv834155604:82 , srv870048486:82 , srv892172798:82 , srv1211016829:83 , srv1268972875:83 , srv1359498632:83 , srv1490780931:83 , srv1712628003:83 , srv250650710:83 , srv445363866:83 , srv647188461:83 , srv149656646:84 , srv1581141289:84 , srv1684671021:84 , srv1829543476:84 , srv602936375:84 , srv646814768:84 , srv890034065:84 , srv1034205543:85 , srv1085134172:85 , srv1160227009:85 , srv169351458:85 , srv1909515908:85 , srv2003481804:85 , srv257575640:85 , srv265169431:85 , srv329236710:85 , srv37490782:85 , srv729590071:85 , srv758456666:85 , srv807145126:85 , srv1029454692:86 , srv1385348286:86 , srv1418568030:86 , srv1484492778:86 , srv1564795762:86 , srv1631511129:86 , srv1643707682:86 , srv1694311817:86 , srv249315381:86 , srv327828967:86 , srv363145627:86 , srv743562353:86 , srv916118954:86 , srv1050525702:87 , srv1117254217:87 , srv1175682344:87 , srv1419961700:87 , srv1685450761:87 , srv1837764886:87 , srv229600832:87 , srv344434943:87 , srv604391702:87 , srv639749905:87 , srv645616016:87 , srv876835412:87 , srv1003006021:88 , srv1147619936:88 , srv1305139784:88 , srv167924700:88 , srv1708320747:88 , srv1877048959:88 , srv1929271820:88 , srv480324940:88 , srv605265885:88 , srv661650747:88 , srv703551040:88 , srv1735214188:89 , srv1829361696:89 , srv463687759:89 , srv49887054:89 , srv702525181:89 , srv783204527:89 , srv1118592712:90 , srv1785109134:90 , srv1932638794:90 , srv2115117753:90 , srv297260296:90 , srv703826232:90 , srv98033623:90 , srv995454128:90 , srv1041925141:91 , srv1364556464:91 , srv1398559989:91 , srv2000257019:91 , srv981509340:91 , srv1429687180:92 , srv729141042:92 , srv1055264409:479 }
2023-09-16 02:53:15,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:53:15,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837764886=47, srv661650747=81, srv480324940=70, srv1160227009=10, srv1484492778=26, srv1490780931=27, srv1385348286=21, srv1685450761=35, srv344434943=65, srv329236710=64, srv604391702=75, srv1643707682=32, srv1684671021=34, srv1175682344=11, srv890034065=94, srv1305139784=17, srv1211016829=12, srv647188461=80, srv1055264409=5, srv1359498632=18, srv497297057=72, srv2115117753=55, srv1631511129=31, srv1029454692=1, srv645616016=78, srv257575640=59, srv1712628003=39, srv445363866=68, srv297260296=61, srv703826232=84, srv49887054=73, srv265169431=60, srv743562353=87, srv1268972875=15, srv639749905=77, srv602936375=74, srv480645640=71, srv169351458=36, srv1377058761=20, srv363145627=66, srv98033623=97, srv463687759=69, srv1085134172=6, srv1291447420=16, srv178873696=43, srv1829543476=46, srv646814768=79, srv892172798=95, srv995454128=99, srv1117254217=7, srv1929271820=51, srv1003006021=0, srv1909515908=50, srv1708320747=38, srv1581141289=30, srv1932638794=52, srv1429687180=25, srv1034205543=2, srv149656646=28, srv1241859614=14, srv1118592712=8, srv729141042=85, srv1791145707=44, srv2003481804=54, srv1050525702=4, srv1564795762=29, srv1871181948=48, srv2000257019=53, srv981509340=98, srv1694311817=37, srv1877048959=49, srv229600832=56, srv605265885=76, srv729590071=86, srv1786029703=42, srv1234383439=13, srv1419961700=24, srv807145126=90, srv250650710=58, srv916118954=96, srv870048486=92, srv167924700=33, srv1785109134=41, srv327828967=63, srv876835412=93, srv37490782=67, srv703551040=83, srv1418568030=23, srv1147619936=9, srv758456666=88, srv304031640=62, srv1398559989=22, srv834155604=91, srv1041925141=3, srv1829361696=45, srv1364556464=19, srv702525181=82, srv249315381=57, srv1735214188=40, srv783204527=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-16 02:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-16 02:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-16 02:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-16 02:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-16 02:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-16 02:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-16 02:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-16 02:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-16 02:53:15,295 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.043778398760458095, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05912189466459749, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.043778398760458095, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:15,295 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-16 02:53:15,295 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1736526528804566, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.043778398760458095, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05912189466459749, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.043778398760458095, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-16 02:53:15,686 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations.  Found a solution that moves 2675 regions; Going from a computed imbalance of 0.1736526528804566 to a new imbalance of 0.012904985394834239. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021052254182399782); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03187376898439346, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2972222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021052254182399782); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:15,702 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1398559989:79 , srv729141042:79 , srv1041925141:81 , srv1785109134:81 , srv1932638794:81 , srv2115117753:81 , srv297260296:81 , srv702525181:81 , srv98033623:81 , srv981509340:81 , srv1735214188:82 , srv2000257019:82 , srv1147619936:83 , srv1364556464:83 , srv1685450761:83 , srv1829361696:83 , srv480324940:83 , srv605265885:83 , srv645616016:83 , srv703826232:83 , srv1003006021:84 , srv1029454692:84 , srv1118592712:84 , srv1305139784:84 , srv1385348286:84 , srv1419961700:84 , srv1643707682:84 , srv167924700:84 , srv1837764886:84 , srv1877048959:84 , srv344434943:84 , srv463687759:84 , srv604391702:84 , srv639749905:84 , srv661650747:84 , srv703551040:84 , srv783204527:84 , srv995454128:84 , srv1418568030:85 , srv1429687180:85 , srv1708320747:85 , srv1929271820:85 , srv229600832:85 , srv249315381:85 , srv329236710:85 , srv1050525702:86 , srv1085134172:86 , srv1117254217:86 , srv1211016829:86 , srv1484492778:86 , srv1631511129:86 , srv169351458:86 , srv2003481804:86 , srv265169431:86 , srv37490782:86 , srv743562353:86 , srv758456666:86 , srv1160227009:87 , srv1564795762:87 , srv1581141289:87 , srv1694311817:87 , srv1909515908:87 , srv257575640:87 , srv49887054:87 , srv807145126:87 , srv876835412:87 , srv916118954:87 , srv1034205543:88 , srv1359498632:88 , srv149656646:88 , srv1791145707:88 , srv1829543476:88 , srv250650710:88 , srv327828967:88 , srv363145627:88 , srv445363866:88 , srv1175682344:89 , srv1684671021:89 , srv480645640:89 , srv647188461:89 , srv729590071:89 , srv1268972875:90 , srv1377058761:90 , srv1712628003:90 , srv304031640:90 , srv497297057:90 , srv602936375:90 , srv646814768:90 , srv890034065:90 , srv892172798:90 , srv1241859614:91 , srv1786029703:91 , srv178873696:91 , srv834155604:91 , srv1291447420:92 , srv1871181948:92 , srv1490780931:93 , srv870048486:93 , srv1234383439:94 , srv1055264409:482 }
2023-09-16 02:53:15,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:53:15,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837764886=47, srv661650747=81, srv480324940=70, srv1160227009=10, srv1484492778=26, srv1490780931=27, srv1385348286=21, srv1685450761=35, srv344434943=65, srv329236710=64, srv604391702=75, srv1643707682=32, srv1684671021=34, srv1175682344=11, srv890034065=94, srv1305139784=17, srv1211016829=12, srv647188461=80, srv1055264409=5, srv1359498632=18, srv497297057=72, srv2115117753=55, srv1631511129=31, srv1029454692=1, srv645616016=78, srv257575640=59, srv1712628003=39, srv445363866=68, srv297260296=61, srv703826232=84, srv49887054=73, srv265169431=60, srv743562353=87, srv1268972875=15, srv639749905=77, srv602936375=74, srv480645640=71, srv169351458=36, srv1377058761=20, srv363145627=66, srv98033623=97, srv463687759=69, srv1085134172=6, srv1291447420=16, srv178873696=43, srv1829543476=46, srv646814768=79, srv892172798=95, srv995454128=99, srv1117254217=7, srv1929271820=51, srv1003006021=0, srv1909515908=50, srv1708320747=38, srv1581141289=30, srv1932638794=52, srv1429687180=25, srv1034205543=2, srv149656646=28, srv1241859614=14, srv1118592712=8, srv729141042=85, srv1791145707=44, srv2003481804=54, srv1050525702=4, srv1564795762=29, srv1871181948=48, srv2000257019=53, srv981509340=98, srv1694311817=37, srv1877048959=49, srv229600832=56, srv605265885=76, srv729590071=86, srv1786029703=42, srv1234383439=13, srv1419961700=24, srv807145126=90, srv250650710=58, srv916118954=96, srv870048486=92, srv167924700=33, srv1785109134=41, srv327828967=63, srv876835412=93, srv37490782=67, srv703551040=83, srv1418568030=23, srv1147619936=9, srv758456666=88, srv304031640=62, srv1398559989=22, srv834155604=91, srv1041925141=3, srv1829361696=45, srv1364556464=19, srv702525181=82, srv249315381=57, srv1735214188=40, srv783204527=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:53:15,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:53:15,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:53:15,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:53:15,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:53:15,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:53:15,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:53:15,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:53:15,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-16 02:53:15,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-16 02:53:15,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-16 02:53:15,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-16 02:53:15,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-16 02:53:15,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-16 02:53:15,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-16 02:53:15,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-16 02:53:15,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-16 02:53:15,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-16 02:53:15,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-16 02:53:15,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-16 02:53:15,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-16 02:53:15,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-16 02:53:15,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-16 02:53:15,923 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021052254182399782); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03187376898439346, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021052254182399782); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:15,923 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-16 02:53:15,923 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012886254644493887, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021052254182399782); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03187376898439346, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021052254182399782); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-16 02:53:16,313 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 531 ms to try 5000 different iterations.  Found a solution that moves 1079 regions; Going from a computed imbalance of 0.012886254644493887 to a new imbalance of 4.584708086297598E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005552187531265666); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005898505584103609); MoveCostFunction : (multiplier=7.0, imbalance=0.11988888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005552187531265666); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:16,319 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1429687180:81 , srv1490780931:81 , srv49887054:82 , srv870048486:82 , srv1175682344:83 , srv363145627:83 , srv646814768:83 , srv890034065:83 , srv995454128:83 , srv1050525702:84 , srv1118592712:84 , srv1234383439:84 , srv1268972875:84 , srv1364556464:84 , srv149656646:84 , srv1564795762:84 , srv1694311817:84 , srv1708320747:84 , srv178873696:84 , srv1929271820:84 , srv602936375:84 , srv729590071:84 , srv834155604:84 , srv876835412:84 , srv1034205543:85 , srv1241859614:85 , srv1291447420:85 , srv1305139784:85 , srv1484492778:85 , srv1631511129:85 , srv167924700:85 , srv1684671021:85 , srv1712628003:85 , srv1786029703:85 , srv1829361696:85 , srv1877048959:85 , srv2000257019:85 , srv257575640:85 , srv304031640:85 , srv327828967:85 , srv661650747:85 , srv703551040:85 , srv703826232:85 , srv743562353:85 , srv783204527:85 , srv807145126:85 , srv916118954:85 , srv1003006021:86 , srv1085134172:86 , srv1117254217:86 , srv1160227009:86 , srv1359498632:86 , srv1377058761:86 , srv1419961700:86 , srv1581141289:86 , srv169351458:86 , srv1785109134:86 , srv1829543476:86 , srv1837764886:86 , srv1871181948:86 , srv1909515908:86 , srv1932638794:86 , srv2115117753:86 , srv229600832:86 , srv249315381:86 , srv250650710:86 , srv265169431:86 , srv297260296:86 , srv344434943:86 , srv37490782:86 , srv463687759:86 , srv480324940:86 , srv480645640:86 , srv497297057:86 , srv605265885:86 , srv639749905:86 , srv645616016:86 , srv647188461:86 , srv892172798:86 , srv981509340:86 , srv1029454692:87 , srv1041925141:87 , srv1385348286:87 , srv1398559989:87 , srv1418568030:87 , srv1685450761:87 , srv1735214188:87 , srv2003481804:87 , srv445363866:87 , srv604391702:87 , srv702525181:87 , srv729141042:87 , srv758456666:87 , srv98033623:87 , srv1147619936:88 , srv1211016829:88 , srv1643707682:88 , srv1791145707:88 , srv329236710:90 , srv1055264409:546 }
2023-09-16 02:53:16,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:53:16,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837764886=47, srv661650747=81, srv480324940=70, srv1160227009=10, srv1484492778=26, srv1490780931=27, srv1385348286=21, srv1685450761=35, srv344434943=65, srv329236710=64, srv604391702=75, srv1643707682=32, srv1684671021=34, srv1175682344=11, srv890034065=94, srv1305139784=17, srv1211016829=12, srv647188461=80, srv1055264409=5, srv1359498632=18, srv497297057=72, srv2115117753=55, srv1631511129=31, srv1029454692=1, srv645616016=78, srv257575640=59, srv1712628003=39, srv445363866=68, srv297260296=61, srv703826232=84, srv49887054=73, srv265169431=60, srv743562353=87, srv1268972875=15, srv639749905=77, srv602936375=74, srv480645640=71, srv169351458=36, srv1377058761=20, srv363145627=66, srv98033623=97, srv463687759=69, srv1085134172=6, srv1291447420=16, srv178873696=43, srv1829543476=46, srv646814768=79, srv892172798=95, srv995454128=99, srv1117254217=7, srv1929271820=51, srv1003006021=0, srv1909515908=50, srv1708320747=38, srv1581141289=30, srv1932638794=52, srv1429687180=25, srv1034205543=2, srv149656646=28, srv1241859614=14, srv1118592712=8, srv729141042=85, srv1791145707=44, srv2003481804=54, srv1050525702=4, srv1564795762=29, srv1871181948=48, srv2000257019=53, srv981509340=98, srv1694311817=37, srv1877048959=49, srv229600832=56, srv605265885=76, srv729590071=86, srv1786029703=42, srv1234383439=13, srv1419961700=24, srv807145126=90, srv250650710=58, srv916118954=96, srv870048486=92, srv167924700=33, srv1785109134=41, srv327828967=63, srv876835412=93, srv37490782=67, srv703551040=83, srv1418568030=23, srv1147619936=9, srv758456666=88, srv304031640=62, srv1398559989=22, srv834155604=91, srv1041925141=3, srv1829361696=45, srv1364556464=19, srv702525181=82, srv249315381=57, srv1735214188=40, srv783204527=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:53:16,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:53:16,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:53:16,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:53:16,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:53:16,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:53:16,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:53:16,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-16 02:53:16,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-16 02:53:16,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-16 02:53:16,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-16 02:53:16,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-16 02:53:16,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-16 02:53:16,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-16 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-16 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-16 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-16 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-16 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-16 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-16 02:53:16,501 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005552187531265666); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005898505584103609); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005552187531265666); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:16,501 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-16 02:53:16,501 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.509154891466785E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005552187531265666); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005898505584103609); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005552187531265666); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-16 02:53:16,897 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 533 ms to try 5000 different iterations.  Found a solution that moves 262 regions; Going from a computed imbalance of 4.509154891466785E-4 to a new imbalance of 2.1155712762896078E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.029111111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:16,898 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv329236710:81 , srv1147619936:82 , srv1041925141:83 , srv1117254217:83 , srv1377058761:83 , srv2000257019:83 , srv463687759:83 , srv703826232:83 , srv876835412:83 , srv1003006021:84 , srv1034205543:84 , srv1118592712:84 , srv1160227009:84 , srv1175682344:84 , srv1211016829:84 , srv1234383439:84 , srv1241859614:84 , srv1291447420:84 , srv1364556464:84 , srv1418568030:84 , srv1429687180:84 , srv1684671021:84 , srv1735214188:84 , srv1786029703:84 , srv1791145707:84 , srv1829543476:84 , srv1871181948:84 , srv1909515908:84 , srv1932638794:84 , srv2003481804:84 , srv229600832:84 , srv327828967:84 , srv445363866:84 , srv497297057:84 , srv49887054:84 , srv602936375:84 , srv604391702:84 , srv647188461:84 , srv729141042:84 , srv729590071:84 , srv758456666:84 , srv783204527:84 , srv892172798:84 , srv916118954:84 , srv98033623:84 , srv981509340:84 , srv1029454692:85 , srv1050525702:85 , srv1085134172:85 , srv1268972875:85 , srv1305139784:85 , srv1359498632:85 , srv1385348286:85 , srv1419961700:85 , srv1484492778:85 , srv1490780931:85 , srv149656646:85 , srv1564795762:85 , srv1581141289:85 , srv1631511129:85 , srv1643707682:85 , srv167924700:85 , srv1685450761:85 , srv169351458:85 , srv1694311817:85 , srv1708320747:85 , srv1712628003:85 , srv1785109134:85 , srv178873696:85 , srv1829361696:85 , srv1837764886:85 , srv1877048959:85 , srv1929271820:85 , srv2115117753:85 , srv249315381:85 , srv250650710:85 , srv257575640:85 , srv265169431:85 , srv297260296:85 , srv344434943:85 , srv37490782:85 , srv480645640:85 , srv605265885:85 , srv639749905:85 , srv661650747:85 , srv702525181:85 , srv703551040:85 , srv807145126:85 , srv870048486:85 , srv890034065:85 , srv995454128:85 , srv1398559989:86 , srv304031640:86 , srv363145627:86 , srv480324940:86 , srv645616016:86 , srv646814768:86 , srv743562353:86 , srv834155604:86 , srv1055264409:635 }
2023-09-16 02:53:16,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:53:16,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837764886=47, srv661650747=81, srv480324940=70, srv1160227009=10, srv1484492778=26, srv1490780931=27, srv1385348286=21, srv1685450761=35, srv344434943=65, srv329236710=64, srv604391702=75, srv1643707682=32, srv1684671021=34, srv1175682344=11, srv890034065=94, srv1305139784=17, srv1211016829=12, srv647188461=80, srv1055264409=5, srv1359498632=18, srv497297057=72, srv2115117753=55, srv1631511129=31, srv1029454692=1, srv645616016=78, srv257575640=59, srv1712628003=39, srv445363866=68, srv297260296=61, srv703826232=84, srv49887054=73, srv265169431=60, srv743562353=87, srv1268972875=15, srv639749905=77, srv602936375=74, srv480645640=71, srv169351458=36, srv1377058761=20, srv363145627=66, srv98033623=97, srv463687759=69, srv1085134172=6, srv1291447420=16, srv178873696=43, srv1829543476=46, srv646814768=79, srv892172798=95, srv995454128=99, srv1117254217=7, srv1929271820=51, srv1003006021=0, srv1909515908=50, srv1708320747=38, srv1581141289=30, srv1932638794=52, srv1429687180=25, srv1034205543=2, srv149656646=28, srv1241859614=14, srv1118592712=8, srv729141042=85, srv1791145707=44, srv2003481804=54, srv1050525702=4, srv1564795762=29, srv1871181948=48, srv2000257019=53, srv981509340=98, srv1694311817=37, srv1877048959=49, srv229600832=56, srv605265885=76, srv729590071=86, srv1786029703=42, srv1234383439=13, srv1419961700=24, srv807145126=90, srv250650710=58, srv916118954=96, srv870048486=92, srv167924700=33, srv1785109134=41, srv327828967=63, srv876835412=93, srv37490782=67, srv703551040=83, srv1418568030=23, srv1147619936=9, srv758456666=88, srv304031640=62, srv1398559989=22, srv834155604=91, srv1041925141=3, srv1829361696=45, srv1364556464=19, srv702525181=82, srv249315381=57, srv1735214188=40, srv783204527=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:53:16,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:53:16,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-16 02:53:16,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-16 02:53:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-16 02:53:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-16 02:53:16,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-16 02:53:16,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-16 02:53:16,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-16 02:53:16,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-16 02:53:16,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-16 02:53:16,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-16 02:53:17,032 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.932114955199033E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:17,032 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv329236710:81 , srv1147619936:82 , srv1041925141:83 , srv1117254217:83 , srv1377058761:83 , srv2000257019:83 , srv463687759:83 , srv703826232:83 , srv876835412:83 , srv1003006021:84 , srv1034205543:84 , srv1118592712:84 , srv1160227009:84 , srv1175682344:84 , srv1211016829:84 , srv1234383439:84 , srv1241859614:84 , srv1291447420:84 , srv1364556464:84 , srv1418568030:84 , srv1429687180:84 , srv1684671021:84 , srv1735214188:84 , srv1786029703:84 , srv1791145707:84 , srv1829543476:84 , srv1871181948:84 , srv1909515908:84 , srv1932638794:84 , srv2003481804:84 , srv229600832:84 , srv327828967:84 , srv445363866:84 , srv497297057:84 , srv49887054:84 , srv602936375:84 , srv604391702:84 , srv647188461:84 , srv729141042:84 , srv729590071:84 , srv758456666:84 , srv783204527:84 , srv892172798:84 , srv916118954:84 , srv98033623:84 , srv981509340:84 , srv1029454692:85 , srv1050525702:85 , srv1085134172:85 , srv1268972875:85 , srv1305139784:85 , srv1359498632:85 , srv1385348286:85 , srv1419961700:85 , srv1484492778:85 , srv1490780931:85 , srv149656646:85 , srv1564795762:85 , srv1581141289:85 , srv1631511129:85 , srv1643707682:85 , srv167924700:85 , srv1685450761:85 , srv169351458:85 , srv1694311817:85 , srv1708320747:85 , srv1712628003:85 , srv1785109134:85 , srv178873696:85 , srv1829361696:85 , srv1837764886:85 , srv1877048959:85 , srv1929271820:85 , srv2115117753:85 , srv249315381:85 , srv250650710:85 , srv257575640:85 , srv265169431:85 , srv297260296:85 , srv344434943:85 , srv37490782:85 , srv480645640:85 , srv605265885:85 , srv639749905:85 , srv661650747:85 , srv702525181:85 , srv703551040:85 , srv807145126:85 , srv870048486:85 , srv890034065:85 , srv995454128:85 , srv1398559989:86 , srv304031640:86 , srv363145627:86 , srv480324940:86 , srv645616016:86 , srv646814768:86 , srv743562353:86 , srv834155604:86 , srv1055264409:635 }
2023-09-16 02:53:17,061 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=388 (was 343) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6419 (was 7062)
2023-09-16 02:53:17,069 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=388, ProcessCount=169, AvailableMemoryMB=6418
2023-09-16 02:53:17,069 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-16 02:53:17,070 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-16 02:53:17,077 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1082615501:84 , srv1238780581:84 , srv1352919750:84 , srv1386188519:84 , srv1386258349:84 , srv1497260364:84 , srv150460610:84 , srv1626040656:84 , srv1732112585:84 , srv176706478:84 , srv1779429367:84 , srv1845166751:84 , srv1853165302:84 , srv1898754311:84 , srv1999614409:84 , srv217079175:84 , srv249236533:84 , srv267304211:84 , srv38118939:84 , srv394667833:84 , srv421873709:84 , srv516334613:84 , srv559879908:84 , srv622412743:84 , srv720482464:84 , srv733314367:84 , srv767217751:84 , srv768183888:84 , srv774638229:84 , srv1810573760:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-16 02:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810573760=11, srv267304211=18, srv1082615501=0, srv1732112585=8, srv516334613=22, srv622412743=24, srv1779429367=10, srv1999614409=15, srv217079175=16, srv1626040656=7, srv176706478=9, srv1238780581=1, srv1386188519=3, srv1853165302=13, srv559879908=23, srv249236533=17, srv767217751=27, srv774638229=29, srv733314367=26, srv1845166751=12, srv720482464=25, srv1386258349=4, srv421873709=21, srv1497260364=5, srv150460610=6, srv768183888=28, srv394667833=20, srv1898754311=14, srv38118939=19, srv1352919750=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:53:17,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:53:17,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-16 02:53:17,104 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.096786265940526, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:53:17,104 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912498013627896, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.096786265940526, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-16 02:55:17,091 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5726784 different iterations.  Found a solution that moves 1871 regions; Going from a computed imbalance of 0.9912498013627896 to a new imbalance of 4.367007337919408E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.692962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:55:17,096 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1082615501:90 , srv1238780581:90 , srv1352919750:90 , srv1386188519:90 , srv1386258349:90 , srv1497260364:90 , srv150460610:90 , srv1626040656:90 , srv1732112585:90 , srv176706478:90 , srv1779429367:90 , srv1810573760:90 , srv1845166751:90 , srv1853165302:90 , srv1898754311:90 , srv1999614409:90 , srv217079175:90 , srv249236533:90 , srv267304211:90 , srv38118939:90 , srv394667833:90 , srv421873709:90 , srv516334613:90 , srv559879908:90 , srv622412743:90 , srv720482464:90 , srv733314367:90 , srv767217751:90 , srv768183888:90 , srv774638229:90 }
2023-09-16 02:55:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-16 02:55:17,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810573760=11, srv267304211=18, srv1082615501=0, srv1732112585=8, srv516334613=22, srv622412743=24, srv1779429367=10, srv1999614409=15, srv217079175=16, srv1626040656=7, srv176706478=9, srv1238780581=1, srv1386188519=3, srv1853165302=13, srv559879908=23, srv249236533=17, srv767217751=27, srv774638229=29, srv733314367=26, srv1845166751=12, srv720482464=25, srv1386258349=4, srv421873709=21, srv1497260364=5, srv150460610=6, srv768183888=28, srv394667833=20, srv1898754311=14, srv38118939=19, srv1352919750=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-16 02:55:17,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-16 02:55:17,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-16 02:55:17,115 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-16 02:55:17,116 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1082615501:90 , srv1238780581:90 , srv1352919750:90 , srv1386188519:90 , srv1386258349:90 , srv1497260364:90 , srv150460610:90 , srv1626040656:90 , srv1732112585:90 , srv176706478:90 , srv1779429367:90 , srv1810573760:90 , srv1845166751:90 , srv1853165302:90 , srv1898754311:90 , srv1999614409:90 , srv217079175:90 , srv249236533:90 , srv267304211:90 , srv38118939:90 , srv394667833:90 , srv421873709:90 , srv516334613:90 , srv559879908:90 , srv622412743:90 , srv720482464:90 , srv733314367:90 , srv767217751:90 , srv768183888:90 , srv774638229:90 }
2023-09-16 02:55:17,128 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=151 (was 388), ProcessCount=169 (was 169), AvailableMemoryMB=5610 (was 6418)