2023-09-21 03:53:11,851 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-21 03:53:12,177 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-21 03:53:12,286 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-21 03:53:12,345 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-21 03:53:12,353 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-21 03:53:12,366 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=288, ProcessCount=171, AvailableMemoryMB=4996
2023-09-21 03:53:12,369 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-21 03:53:12,369 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-21 03:53:12,521 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023964872:84 , srv1033113648:84 , srv1059946209:84 , srv1063026411:84 , srv1094655223:84 , srv1107128902:84 , srv1108142548:84 , srv112267043:84 , srv1137720519:84 , srv1148408188:84 , srv1152225001:84 , srv115509920:84 , srv1158866639:84 , srv1159761472:84 , srv1170522162:84 , srv1203329633:84 , srv1207203542:84 , srv1226221327:84 , srv1229846134:84 , srv1232685562:84 , srv1254939989:84 , srv1273692311:84 , srv1289822860:84 , srv1310500680:84 , srv1335975056:84 , srv1361724206:84 , srv1378391245:84 , srv1393761209:84 , srv1426597707:84 , srv1433037853:84 , srv146947149:84 , srv147277717:84 , srv1535496309:84 , srv1555574329:84 , srv1563947248:84 , srv1596428609:84 , srv1624981768:84 , srv1666966110:84 , srv1695817804:84 , srv1704204780:84 , srv1706649444:84 , srv1711972430:84 , srv1731947658:84 , srv1749923897:84 , srv175833963:84 , srv1830160774:84 , srv1839578322:84 , srv1841253452:84 , srv1841843058:84 , srv1841949972:84 , srv184715556:84 , srv1880837125:84 , srv1906415796:84 , srv1911417897:84 , srv191837430:84 , srv1966655398:84 , srv2001018721:84 , srv2034951393:84 , srv2074703941:84 , srv2122679944:84 , srv215317937:84 , srv224065443:84 , srv239998805:84 , srv240336336:84 , srv259952442:84 , srv264981555:84 , srv295995047:84 , srv308630324:84 , srv317816618:84 , srv331321630:84 , srv335179456:84 , srv351737134:84 , srv432181373:84 , srv499775959:84 , srv548541798:84 , srv551978896:84 , srv564285984:84 , srv574190581:84 , srv594882407:84 , srv629357849:84 , srv66065169:84 , srv671760735:84 , srv694193339:84 , srv697668555:84 , srv706635664:84 , srv731814364:84 , srv734113641:84 , srv740406989:84 , srv744190270:84 , srv746394626:84 , srv754908397:84 , srv762373326:84 , srv763451863:84 , srv770697887:84 , srv806496061:84 , srv822249475:84 , srv834092127:84 , srv863775350:84 , srv892237323:84 , srv1076561125:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-21 03:53:12,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:53:12,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074703941=59, srv1711972430=42, srv706635664=85, srv763451863=93, srv746394626=90, srv1148408188=10, srv1841253452=48, srv147277717=32, srv215317937=61, srv66065169=81, srv1152225001=11, srv697668555=84, srv259952442=65, srv1880837125=52, srv1229846134=19, srv308630324=68, srv1335975056=25, srv1624981768=37, srv264981555=66, srv317816618=69, srv564285984=77, srv1207203542=17, srv1107128902=6, srv762373326=92, srv834092127=97, srv1059946209=2, srv115509920=12, srv2001018721=57, srv734113641=87, srv770697887=94, srv1226221327=18, srv335179456=71, srv694193339=83, srv551978896=76, srv1310500680=24, srv1378391245=27, srv1596428609=36, srv1094655223=5, srv1911417897=54, srv239998805=63, srv1232685562=20, srv240336336=64, srv432181373=73, srv1361724206=26, srv1704204780=40, srv1841843058=49, srv822249475=96, srv184715556=51, srv1706649444=41, srv863775350=98, srv1839578322=47, srv1966655398=56, srv574190581=78, srv740406989=88, srv1137720519=9, srv731814364=86, srv175833963=45, srv594882407=79, srv892237323=99, srv1666966110=38, srv1170522162=15, srv629357849=80, srv806496061=95, srv1033113648=1, srv2034951393=58, srv2122679944=60, srv1433037853=30, srv1695817804=39, srv295995047=67, srv1023964872=0, srv1563947248=35, srv499775959=74, srv744190270=89, srv1289822860=23, srv1254939989=21, srv548541798=75, srv146947149=31, srv1108142548=7, srv1426597707=29, srv1749923897=44, srv1393761209=28, srv1841949972=50, srv671760735=82, srv1159761472=14, srv1076561125=4, srv1535496309=33, srv1731947658=43, srv224065443=62, srv1063026411=3, srv1555574329=34, srv112267043=8, srv1830160774=46, srv1906415796=53, srv351737134=72, srv191837430=55, srv1203329633=16, srv1273692311=22, srv331321630=70, srv754908397=91, srv1158866639=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:53:12,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:53:12,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:53:12,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-21 03:53:12,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-21 03:53:12,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-21 03:53:12,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-21 03:53:12,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-21 03:53:12,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-21 03:53:12,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-21 03:53:12,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-21 03:53:12,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-21 03:53:12,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-21 03:53:12,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-21 03:53:12,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-21 03:53:12,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-21 03:53:12,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-21 03:53:12,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-21 03:53:12,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-21 03:53:12,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-21 03:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-21 03:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-21 03:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-21 03:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-21 03:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-21 03:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-21 03:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-21 03:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-21 03:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-21 03:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-21 03:53:12,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:12,860 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-21 03:53:12,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-21 03:53:13,542 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 871 ms to try 5000 different iterations.  Found a solution that moves 4726 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17524754271490597. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039184860576710764, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05432991458125335, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5251111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039184860576710764, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17025, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:13,563 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1966655398:81 , srv1906415796:82 , srv2034951393:82 , srv331321630:82 , srv432181373:82 , srv146947149:83 , srv1695817804:83 , srv1711972430:83 , srv215317937:83 , srv351737134:83 , srv734113641:83 , srv1063026411:84 , srv1107128902:84 , srv1254939989:84 , srv1273692311:84 , srv1426597707:84 , srv317816618:84 , srv551978896:84 , srv564285984:84 , srv744190270:84 , srv770697887:84 , srv822249475:84 , srv1023964872:85 , srv1170522162:85 , srv1361724206:85 , srv1433037853:85 , srv1555574329:85 , srv1666966110:85 , srv1839578322:85 , srv1880837125:85 , srv1911417897:85 , srv239998805:85 , srv240336336:85 , srv295995047:85 , srv308630324:85 , srv548541798:85 , srv594882407:85 , srv629357849:85 , srv694193339:85 , srv746394626:85 , srv806496061:85 , srv1033113648:86 , srv112267043:86 , srv1137720519:86 , srv1159761472:86 , srv1203329633:86 , srv1207203542:86 , srv1289822860:86 , srv1310500680:86 , srv1563947248:86 , srv1706649444:86 , srv1830160774:86 , srv1841843058:86 , srv1841949972:86 , srv2001018721:86 , srv574190581:86 , srv706635664:86 , srv754908397:86 , srv763451863:86 , srv834092127:86 , srv1059946209:87 , srv1094655223:87 , srv1152225001:87 , srv147277717:87 , srv1535496309:87 , srv1596428609:87 , srv1731947658:87 , srv175833963:87 , srv1841253452:87 , srv224065443:87 , srv264981555:87 , srv499775959:87 , srv697668555:87 , srv1108142548:88 , srv1158866639:88 , srv1378391245:88 , srv191837430:88 , srv2122679944:88 , srv335179456:88 , srv892237323:88 , srv731814364:89 , srv762373326:89 , srv1148408188:90 , srv1226221327:90 , srv184715556:90 , srv2074703941:90 , srv671760735:90 , srv115509920:91 , srv259952442:91 , srv1335975056:92 , srv1393761209:92 , srv1704204780:92 , srv1749923897:92 , srv66065169:92 , srv740406989:92 , srv863775350:92 , srv1229846134:93 , srv1232685562:94 , srv1624981768:94 , srv1076561125:438 }
2023-09-21 03:53:13,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:53:13,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074703941=59, srv1711972430=42, srv706635664=85, srv763451863=93, srv746394626=90, srv1148408188=10, srv1841253452=48, srv147277717=32, srv215317937=61, srv66065169=81, srv1152225001=11, srv697668555=84, srv259952442=65, srv1880837125=52, srv1229846134=19, srv308630324=68, srv1335975056=25, srv1624981768=37, srv264981555=66, srv317816618=69, srv564285984=77, srv1207203542=17, srv1107128902=6, srv762373326=92, srv834092127=97, srv1059946209=2, srv115509920=12, srv2001018721=57, srv734113641=87, srv770697887=94, srv1226221327=18, srv335179456=71, srv694193339=83, srv551978896=76, srv1310500680=24, srv1378391245=27, srv1596428609=36, srv1094655223=5, srv1911417897=54, srv239998805=63, srv1232685562=20, srv240336336=64, srv432181373=73, srv1361724206=26, srv1704204780=40, srv1841843058=49, srv822249475=96, srv184715556=51, srv1706649444=41, srv863775350=98, srv1839578322=47, srv1966655398=56, srv574190581=78, srv740406989=88, srv1137720519=9, srv731814364=86, srv175833963=45, srv594882407=79, srv892237323=99, srv1666966110=38, srv1170522162=15, srv629357849=80, srv806496061=95, srv1033113648=1, srv2034951393=58, srv2122679944=60, srv1433037853=30, srv1695817804=39, srv295995047=67, srv1023964872=0, srv1563947248=35, srv499775959=74, srv744190270=89, srv1289822860=23, srv1254939989=21, srv548541798=75, srv146947149=31, srv1108142548=7, srv1426597707=29, srv1749923897=44, srv1393761209=28, srv1841949972=50, srv671760735=82, srv1159761472=14, srv1076561125=4, srv1535496309=33, srv1731947658=43, srv224065443=62, srv1063026411=3, srv1555574329=34, srv112267043=8, srv1830160774=46, srv1906415796=53, srv351737134=72, srv191837430=55, srv1203329633=16, srv1273692311=22, srv331321630=70, srv754908397=91, srv1158866639=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:53:13,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:53:13,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:53:13,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:53:13,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-21 03:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-21 03:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-21 03:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-21 03:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-21 03:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-21 03:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-21 03:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-21 03:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-21 03:53:13,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039184860576710764, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05432991458125335, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039184860576710764, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17025, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:13,785 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-21 03:53:13,785 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1752144505556131, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039184860576710764, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05432991458125335, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039184860576710764, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17025, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-21 03:53:14,159 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 536 ms to try 5000 different iterations.  Found a solution that moves 2600 regions; Going from a computed imbalance of 0.1752144505556131 to a new imbalance of 0.015162713875521504. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021709000629180176); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036184463184936747, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021709000629180176); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:14,174 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1624981768:78 , srv1232685562:79 , srv1704204780:79 , srv115509920:80 , srv1229846134:80 , srv1749923897:80 , srv184715556:80 , srv1335975056:81 , srv762373326:81 , srv1148408188:82 , srv1226221327:82 , srv1393761209:82 , srv66065169:82 , srv740406989:82 , srv863775350:82 , srv1378391245:83 , srv147277717:83 , srv1596428609:83 , srv259952442:83 , srv335179456:83 , srv671760735:83 , srv697668555:83 , srv731814364:83 , srv892237323:83 , srv112267043:84 , srv1152225001:84 , srv1207203542:84 , srv1289822860:84 , srv1535496309:84 , srv1706649444:84 , srv175833963:84 , srv2074703941:84 , srv2122679944:84 , srv1059946209:85 , srv1108142548:85 , srv1159761472:85 , srv1731947658:85 , srv1841949972:85 , srv191837430:85 , srv499775959:85 , srv763451863:85 , srv1023964872:86 , srv1033113648:86 , srv1203329633:86 , srv1310500680:86 , srv1563947248:86 , srv1666966110:86 , srv1830160774:86 , srv1841253452:86 , srv1841843058:86 , srv1911417897:86 , srv2001018721:86 , srv239998805:86 , srv264981555:86 , srv295995047:86 , srv308630324:86 , srv706635664:86 , srv754908397:86 , srv1063026411:87 , srv1094655223:87 , srv1107128902:87 , srv1158866639:87 , srv1170522162:87 , srv1426597707:87 , srv1433037853:87 , srv1695817804:87 , srv1880837125:87 , srv240336336:87 , srv317816618:87 , srv594882407:87 , srv694193339:87 , srv806496061:87 , srv834092127:87 , srv1361724206:88 , srv1839578322:88 , srv224065443:88 , srv331321630:88 , srv548541798:88 , srv551978896:88 , srv564285984:88 , srv629357849:88 , srv734113641:88 , srv744190270:88 , srv746394626:88 , srv770697887:88 , srv822249475:88 , srv1254939989:89 , srv1273692311:89 , srv146947149:89 , srv1555574329:89 , srv1711972430:89 , srv2034951393:89 , srv215317937:89 , srv574190581:89 , srv1137720519:90 , srv432181373:90 , srv1906415796:91 , srv351737134:91 , srv1966655398:93 , srv1076561125:529 }
2023-09-21 03:53:14,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:53:14,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074703941=59, srv1711972430=42, srv706635664=85, srv763451863=93, srv746394626=90, srv1148408188=10, srv1841253452=48, srv147277717=32, srv215317937=61, srv66065169=81, srv1152225001=11, srv697668555=84, srv259952442=65, srv1880837125=52, srv1229846134=19, srv308630324=68, srv1335975056=25, srv1624981768=37, srv264981555=66, srv317816618=69, srv564285984=77, srv1207203542=17, srv1107128902=6, srv762373326=92, srv834092127=97, srv1059946209=2, srv115509920=12, srv2001018721=57, srv734113641=87, srv770697887=94, srv1226221327=18, srv335179456=71, srv694193339=83, srv551978896=76, srv1310500680=24, srv1378391245=27, srv1596428609=36, srv1094655223=5, srv1911417897=54, srv239998805=63, srv1232685562=20, srv240336336=64, srv432181373=73, srv1361724206=26, srv1704204780=40, srv1841843058=49, srv822249475=96, srv184715556=51, srv1706649444=41, srv863775350=98, srv1839578322=47, srv1966655398=56, srv574190581=78, srv740406989=88, srv1137720519=9, srv731814364=86, srv175833963=45, srv594882407=79, srv892237323=99, srv1666966110=38, srv1170522162=15, srv629357849=80, srv806496061=95, srv1033113648=1, srv2034951393=58, srv2122679944=60, srv1433037853=30, srv1695817804=39, srv295995047=67, srv1023964872=0, srv1563947248=35, srv499775959=74, srv744190270=89, srv1289822860=23, srv1254939989=21, srv548541798=75, srv146947149=31, srv1108142548=7, srv1426597707=29, srv1749923897=44, srv1393761209=28, srv1841949972=50, srv671760735=82, srv1159761472=14, srv1076561125=4, srv1535496309=33, srv1731947658=43, srv224065443=62, srv1063026411=3, srv1555574329=34, srv112267043=8, srv1830160774=46, srv1906415796=53, srv351737134=72, srv191837430=55, srv1203329633=16, srv1273692311=22, srv331321630=70, srv754908397=91, srv1158866639=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:53:14,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:53:14,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:53:14,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:53:14,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:53:14,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-21 03:53:14,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-21 03:53:14,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-21 03:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-21 03:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-21 03:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-21 03:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-21 03:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-21 03:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-21 03:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-21 03:53:14,357 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021709000629180176); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036184463184936747, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021709000629180176); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:14,357 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-21 03:53:14,358 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015144508286405645, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021709000629180176); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036184463184936747, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021709000629180176); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-21 03:53:14,657 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 409 ms to try 5000 different iterations.  Found a solution that moves 1015 regions; Going from a computed imbalance of 0.015144508286405645 to a new imbalance of 7.460823418662195E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009904142134751739); MoveCostFunction : (multiplier=7.0, imbalance=0.11277777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:14,659 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1137720519:82 , srv1158866639:82 , srv2074703941:83 , srv224065443:83 , srv259952442:83 , srv351737134:83 , srv1094655223:84 , srv1229846134:84 , srv1232685562:84 , srv1335975056:84 , srv1393761209:84 , srv1555574329:84 , srv1839578322:84 , srv191837430:84 , srv1966655398:84 , srv2122679944:84 , srv264981555:84 , srv548541798:84 , srv574190581:84 , srv629357849:84 , srv66065169:84 , srv740406989:84 , srv863775350:84 , srv1059946209:85 , srv1108142548:85 , srv1170522162:85 , srv1254939989:85 , srv1273692311:85 , srv1310500680:85 , srv1361724206:85 , srv146947149:85 , srv1624981768:85 , srv1731947658:85 , srv1830160774:85 , srv1841843058:85 , srv1880837125:85 , srv1906415796:85 , srv2001018721:85 , srv215317937:85 , srv432181373:85 , srv551978896:85 , srv594882407:85 , srv671760735:85 , srv694193339:85 , srv744190270:85 , srv770697887:85 , srv834092127:85 , srv1023964872:86 , srv1033113648:86 , srv1148408188:86 , srv1152225001:86 , srv115509920:86 , srv1159761472:86 , srv1203329633:86 , srv1226221327:86 , srv1433037853:86 , srv1535496309:86 , srv1563947248:86 , srv1666966110:86 , srv1711972430:86 , srv1749923897:86 , srv1841253452:86 , srv1841949972:86 , srv1911417897:86 , srv2034951393:86 , srv239998805:86 , srv240336336:86 , srv295995047:86 , srv499775959:86 , srv564285984:86 , srv706635664:86 , srv731814364:86 , srv734113641:86 , srv746394626:86 , srv754908397:86 , srv763451863:86 , srv892237323:86 , srv1063026411:87 , srv1107128902:87 , srv112267043:87 , srv1207203542:87 , srv1289822860:87 , srv1378391245:87 , srv1426597707:87 , srv1596428609:87 , srv1704204780:87 , srv1706649444:87 , srv175833963:87 , srv184715556:87 , srv308630324:87 , srv317816618:87 , srv331321630:87 , srv335179456:87 , srv697668555:87 , srv762373326:87 , srv806496061:87 , srv822249475:87 , srv147277717:88 , srv1695817804:88 , srv1076561125:540 }
2023-09-21 03:53:14,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:53:14,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074703941=59, srv1711972430=42, srv706635664=85, srv763451863=93, srv746394626=90, srv1148408188=10, srv1841253452=48, srv147277717=32, srv215317937=61, srv66065169=81, srv1152225001=11, srv697668555=84, srv259952442=65, srv1880837125=52, srv1229846134=19, srv308630324=68, srv1335975056=25, srv1624981768=37, srv264981555=66, srv317816618=69, srv564285984=77, srv1207203542=17, srv1107128902=6, srv762373326=92, srv834092127=97, srv1059946209=2, srv115509920=12, srv2001018721=57, srv734113641=87, srv770697887=94, srv1226221327=18, srv335179456=71, srv694193339=83, srv551978896=76, srv1310500680=24, srv1378391245=27, srv1596428609=36, srv1094655223=5, srv1911417897=54, srv239998805=63, srv1232685562=20, srv240336336=64, srv432181373=73, srv1361724206=26, srv1704204780=40, srv1841843058=49, srv822249475=96, srv184715556=51, srv1706649444=41, srv863775350=98, srv1839578322=47, srv1966655398=56, srv574190581=78, srv740406989=88, srv1137720519=9, srv731814364=86, srv175833963=45, srv594882407=79, srv892237323=99, srv1666966110=38, srv1170522162=15, srv629357849=80, srv806496061=95, srv1033113648=1, srv2034951393=58, srv2122679944=60, srv1433037853=30, srv1695817804=39, srv295995047=67, srv1023964872=0, srv1563947248=35, srv499775959=74, srv744190270=89, srv1289822860=23, srv1254939989=21, srv548541798=75, srv146947149=31, srv1108142548=7, srv1426597707=29, srv1749923897=44, srv1393761209=28, srv1841949972=50, srv671760735=82, srv1159761472=14, srv1076561125=4, srv1535496309=33, srv1731947658=43, srv224065443=62, srv1063026411=3, srv1555574329=34, srv112267043=8, srv1830160774=46, srv1906415796=53, srv351737134=72, srv191837430=55, srv1203329633=16, srv1273692311=22, srv331321630=70, srv754908397=91, srv1158866639=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:53:14,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:53:14,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:53:14,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:53:14,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:53:14,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:53:14,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:53:14,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:53:14,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-21 03:53:14,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-21 03:53:14,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-21 03:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-21 03:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-21 03:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-21 03:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-21 03:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-21 03:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-21 03:53:14,802 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009904142134751739); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:14,802 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-21 03:53:14,803 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.389751599613746E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009904142134751739); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-21 03:53:15,154 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations.  Found a solution that moves 255 regions; Going from a computed imbalance of 7.389751599613746E-4 to a new imbalance of 1.286068662571445E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.028333333333333332, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:15,155 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1108142548:83 , srv1393761209:83 , srv1841253452:83 , srv1966655398:83 , srv240336336:83 , srv574190581:83 , srv746394626:83 , srv806496061:83 , srv1033113648:84 , srv1094655223:84 , srv1107128902:84 , srv1137720519:84 , srv1148408188:84 , srv1203329633:84 , srv1226221327:84 , srv1254939989:84 , srv1273692311:84 , srv1378391245:84 , srv1426597707:84 , srv1433037853:84 , srv146947149:84 , srv147277717:84 , srv1555574329:84 , srv1563947248:84 , srv1695817804:84 , srv1704204780:84 , srv1711972430:84 , srv1749923897:84 , srv175833963:84 , srv1839578322:84 , srv1906415796:84 , srv239998805:84 , srv308630324:84 , srv317816618:84 , srv335179456:84 , srv432181373:84 , srv564285984:84 , srv66065169:84 , srv671760735:84 , srv706635664:84 , srv731814364:84 , srv740406989:84 , srv754908397:84 , srv770697887:84 , srv822249475:84 , srv834092127:84 , srv863775350:84 , srv1023964872:85 , srv1059946209:85 , srv1063026411:85 , srv112267043:85 , srv1152225001:85 , srv115509920:85 , srv1158866639:85 , srv1159761472:85 , srv1170522162:85 , srv1207203542:85 , srv1229846134:85 , srv1232685562:85 , srv1310500680:85 , srv1335975056:85 , srv1361724206:85 , srv1535496309:85 , srv1596428609:85 , srv1666966110:85 , srv1706649444:85 , srv1731947658:85 , srv1830160774:85 , srv1841843058:85 , srv1841949972:85 , srv184715556:85 , srv1880837125:85 , srv1911417897:85 , srv191837430:85 , srv2001018721:85 , srv2034951393:85 , srv224065443:85 , srv259952442:85 , srv264981555:85 , srv295995047:85 , srv331321630:85 , srv351737134:85 , srv499775959:85 , srv548541798:85 , srv551978896:85 , srv594882407:85 , srv629357849:85 , srv694193339:85 , srv697668555:85 , srv734113641:85 , srv744190270:85 , srv762373326:85 , srv763451863:85 , srv892237323:85 , srv1289822860:86 , srv1624981768:86 , srv2074703941:86 , srv2122679944:86 , srv215317937:86 , srv1076561125:635 }
2023-09-21 03:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074703941=59, srv1711972430=42, srv706635664=85, srv763451863=93, srv746394626=90, srv1148408188=10, srv1841253452=48, srv147277717=32, srv215317937=61, srv66065169=81, srv1152225001=11, srv697668555=84, srv259952442=65, srv1880837125=52, srv1229846134=19, srv308630324=68, srv1335975056=25, srv1624981768=37, srv264981555=66, srv317816618=69, srv564285984=77, srv1207203542=17, srv1107128902=6, srv762373326=92, srv834092127=97, srv1059946209=2, srv115509920=12, srv2001018721=57, srv734113641=87, srv770697887=94, srv1226221327=18, srv335179456=71, srv694193339=83, srv551978896=76, srv1310500680=24, srv1378391245=27, srv1596428609=36, srv1094655223=5, srv1911417897=54, srv239998805=63, srv1232685562=20, srv240336336=64, srv432181373=73, srv1361724206=26, srv1704204780=40, srv1841843058=49, srv822249475=96, srv184715556=51, srv1706649444=41, srv863775350=98, srv1839578322=47, srv1966655398=56, srv574190581=78, srv740406989=88, srv1137720519=9, srv731814364=86, srv175833963=45, srv594882407=79, srv892237323=99, srv1666966110=38, srv1170522162=15, srv629357849=80, srv806496061=95, srv1033113648=1, srv2034951393=58, srv2122679944=60, srv1433037853=30, srv1695817804=39, srv295995047=67, srv1023964872=0, srv1563947248=35, srv499775959=74, srv744190270=89, srv1289822860=23, srv1254939989=21, srv548541798=75, srv146947149=31, srv1108142548=7, srv1426597707=29, srv1749923897=44, srv1393761209=28, srv1841949972=50, srv671760735=82, srv1159761472=14, srv1076561125=4, srv1535496309=33, srv1731947658=43, srv224065443=62, srv1063026411=3, srv1555574329=34, srv112267043=8, srv1830160774=46, srv1906415796=53, srv351737134=72, srv191837430=55, srv1203329633=16, srv1273692311=22, srv331321630=70, srv754908397=91, srv1158866639=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:53:15,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-21 03:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-21 03:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-21 03:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-21 03:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-21 03:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-21 03:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-21 03:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-21 03:53:15,286 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1075138462428321E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:15,286 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1108142548:83 , srv1393761209:83 , srv1841253452:83 , srv1966655398:83 , srv240336336:83 , srv574190581:83 , srv746394626:83 , srv806496061:83 , srv1033113648:84 , srv1094655223:84 , srv1107128902:84 , srv1137720519:84 , srv1148408188:84 , srv1203329633:84 , srv1226221327:84 , srv1254939989:84 , srv1273692311:84 , srv1378391245:84 , srv1426597707:84 , srv1433037853:84 , srv146947149:84 , srv147277717:84 , srv1555574329:84 , srv1563947248:84 , srv1695817804:84 , srv1704204780:84 , srv1711972430:84 , srv1749923897:84 , srv175833963:84 , srv1839578322:84 , srv1906415796:84 , srv239998805:84 , srv308630324:84 , srv317816618:84 , srv335179456:84 , srv432181373:84 , srv564285984:84 , srv66065169:84 , srv671760735:84 , srv706635664:84 , srv731814364:84 , srv740406989:84 , srv754908397:84 , srv770697887:84 , srv822249475:84 , srv834092127:84 , srv863775350:84 , srv1023964872:85 , srv1059946209:85 , srv1063026411:85 , srv112267043:85 , srv1152225001:85 , srv115509920:85 , srv1158866639:85 , srv1159761472:85 , srv1170522162:85 , srv1207203542:85 , srv1229846134:85 , srv1232685562:85 , srv1310500680:85 , srv1335975056:85 , srv1361724206:85 , srv1535496309:85 , srv1596428609:85 , srv1666966110:85 , srv1706649444:85 , srv1731947658:85 , srv1830160774:85 , srv1841843058:85 , srv1841949972:85 , srv184715556:85 , srv1880837125:85 , srv1911417897:85 , srv191837430:85 , srv2001018721:85 , srv2034951393:85 , srv224065443:85 , srv259952442:85 , srv264981555:85 , srv295995047:85 , srv331321630:85 , srv351737134:85 , srv499775959:85 , srv548541798:85 , srv551978896:85 , srv594882407:85 , srv629357849:85 , srv694193339:85 , srv697668555:85 , srv734113641:85 , srv744190270:85 , srv762373326:85 , srv763451863:85 , srv892237323:85 , srv1289822860:86 , srv1624981768:86 , srv2074703941:86 , srv2122679944:86 , srv215317937:86 , srv1076561125:635 }
2023-09-21 03:53:15,314 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=329 (was 288) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=4361 (was 4996)
2023-09-21 03:53:15,321 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=171, AvailableMemoryMB=4361
2023-09-21 03:53:15,322 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-21 03:53:15,322 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-21 03:53:15,336 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv12049820:84 , srv1306454134:84 , srv1333461591:84 , srv1401389701:84 , srv1443737488:84 , srv1516726696:84 , srv1537788776:84 , srv1669869150:84 , srv1682509145:84 , srv1819007753:84 , srv1853503136:84 , srv2001734648:84 , srv2004433126:84 , srv2095822530:84 , srv2134109105:84 , srv2145424373:84 , srv508535766:84 , srv533117730:84 , srv605981095:84 , srv64378208:84 , srv68779956:84 , srv695210435:84 , srv721587048:84 , srv749016030:84 , srv810056895:84 , srv813982209:84 , srv819485133:84 , srv854698435:84 , srv97979214:84 , srv1578329714:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-21 03:53:15,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:53:15,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv721587048=23, srv1516726696=5, srv2001734648=12, srv1682509145=9, srv1537788776=6, srv1401389701=3, srv1853503136=11, srv810056895=25, srv68779956=21, srv12049820=0, srv97979214=29, srv1669869150=8, srv533117730=18, srv749016030=24, srv2134109105=15, srv854698435=28, srv2145424373=16, srv1306454134=1, srv1443737488=4, srv1578329714=7, srv1819007753=10, srv2095822530=14, srv1333461591=2, srv508535766=17, srv813982209=26, srv2004433126=13, srv695210435=22, srv819485133=27, srv64378208=20, srv605981095=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-21 03:53:15,377 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1433556224570554, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:53:15,377 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991264475214965, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1433556224570554, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-21 03:55:15,359 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5683156 different iterations.  Found a solution that moves 1859 regions; Going from a computed imbalance of 0.991264475214965 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:55:15,365 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv12049820:90 , srv1306454134:90 , srv1333461591:90 , srv1401389701:90 , srv1443737488:90 , srv1516726696:90 , srv1537788776:90 , srv1578329714:90 , srv1669869150:90 , srv1682509145:90 , srv1819007753:90 , srv1853503136:90 , srv2001734648:90 , srv2004433126:90 , srv2095822530:90 , srv2134109105:90 , srv2145424373:90 , srv508535766:90 , srv533117730:90 , srv605981095:90 , srv64378208:90 , srv68779956:90 , srv695210435:90 , srv721587048:90 , srv749016030:90 , srv810056895:90 , srv813982209:90 , srv819485133:90 , srv854698435:90 , srv97979214:90 }
2023-09-21 03:55:15,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-21 03:55:15,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv721587048=23, srv1516726696=5, srv2001734648=12, srv1682509145=9, srv1537788776=6, srv1401389701=3, srv1853503136=11, srv810056895=25, srv68779956=21, srv12049820=0, srv97979214=29, srv1669869150=8, srv533117730=18, srv749016030=24, srv2134109105=15, srv854698435=28, srv2145424373=16, srv1306454134=1, srv1443737488=4, srv1578329714=7, srv1819007753=10, srv2095822530=14, srv1333461591=2, srv508535766=17, srv813982209=26, srv2004433126=13, srv695210435=22, srv819485133=27, srv64378208=20, srv605981095=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-21 03:55:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-21 03:55:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-21 03:55:15,384 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-21 03:55:15,384 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv12049820:90 , srv1306454134:90 , srv1333461591:90 , srv1401389701:90 , srv1443737488:90 , srv1516726696:90 , srv1537788776:90 , srv1578329714:90 , srv1669869150:90 , srv1682509145:90 , srv1819007753:90 , srv1853503136:90 , srv2001734648:90 , srv2004433126:90 , srv2095822530:90 , srv2134109105:90 , srv2145424373:90 , srv508535766:90 , srv533117730:90 , srv605981095:90 , srv64378208:90 , srv68779956:90 , srv695210435:90 , srv721587048:90 , srv749016030:90 , srv810056895:90 , srv813982209:90 , srv819485133:90 , srv854698435:90 , srv97979214:90 }
2023-09-21 03:55:15,397 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=150 (was 329), ProcessCount=167 (was 171), AvailableMemoryMB=3599 (was 4361)