2023-09-27 03:53:01,490 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-27 03:53:01,844 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-27 03:53:01,984 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-27 03:53:02,061 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-27 03:53:02,068 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-27 03:53:02,083 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=623, ProcessCount=180, AvailableMemoryMB=3202
2023-09-27 03:53:02,085 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-27 03:53:02,085 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-27 03:53:02,227 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010890228:84 , srv1043523014:84 , srv104926224:84 , srv1059202006:84 , srv1059643373:84 , srv1094918969:84 , srv110894546:84 , srv1112984256:84 , srv114333048:84 , srv1150043550:84 , srv1153436788:84 , srv1191087995:84 , srv1198227000:84 , srv1217467922:84 , srv1218232710:84 , srv1227928115:84 , srv1242254884:84 , srv1265261864:84 , srv1303796180:84 , srv1350920555:84 , srv1369826944:84 , srv1405668770:84 , srv1417002278:84 , srv1447531311:84 , srv1484251986:84 , srv149323629:84 , srv1512019402:84 , srv1520984544:84 , srv1529106056:84 , srv1553176272:84 , srv1612140197:84 , srv1618541007:84 , srv1624086060:84 , srv1645822630:84 , srv1651510569:84 , srv1668521171:84 , srv1676534302:84 , srv1724359191:84 , srv1731829598:84 , srv1742913167:84 , srv1745944480:84 , srv1753567881:84 , srv1763474207:84 , srv1817424126:84 , srv1839830594:84 , srv1866939687:84 , srv1891159215:84 , srv1896538800:84 , srv1915231298:84 , srv192458273:84 , srv1932310283:84 , srv1955155648:84 , srv1972325779:84 , srv1972617554:84 , srv2023963006:84 , srv204822900:84 , srv2055027646:84 , srv2081510179:84 , srv2081959520:84 , srv2097235694:84 , srv211402102:84 , srv2123130779:84 , srv2143574828:84 , srv215621063:84 , srv231522842:84 , srv23805227:84 , srv342031718:84 , srv393429428:84 , srv40189880:84 , srv443439896:84 , srv463133369:84 , srv532694293:84 , srv542257019:84 , srv547079489:84 , srv591383002:84 , srv593687931:84 , srv598224544:84 , srv605857428:84 , srv611908973:84 , srv614294804:84 , srv633011247:84 , srv656340696:84 , srv656842104:84 , srv681538683:84 , srv70784319:84 , srv7272037:84 , srv736483397:84 , srv826093524:84 , srv828372474:84 , srv843243797:84 , srv864088695:84 , srv869571398:84 , srv882043855:84 , srv882802978:84 , srv882892457:84 , srv90372414:84 , srv923684435:84 , srv961343931:84 , srv961868291:84 , srv989142683:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-27 03:53:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:53:02,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1150043550=9, srv1405668770=21, srv104926224=2, srv882802978=93, srv1932310283=50, srv2023963006=54, srv614294804=79, srv1731829598=38, srv1227928115=15, srv1217467922=13, srv2081959520=58, srv1753567881=41, srv542257019=72, srv1645822630=33, srv463133369=70, srv1265261864=17, srv1447531311=23, srv828372474=88, srv1191087995=11, srv2055027646=56, srv591383002=74, srv1010890228=0, srv7272037=85, srv1896538800=47, srv2097235694=59, srv23805227=65, srv826093524=87, srv1198227000=12, srv1972325779=52, srv633011247=80, srv1763474207=42, srv1242254884=16, srv1350920555=19, srv532694293=71, srv211402102=60, srv231522842=64, srv1520984544=27, srv1218232710=14, srv1724359191=37, srv1153436788=10, srv1839830594=44, srv1369826944=20, srv2123130779=61, srv393429428=67, srv611908973=78, srv443439896=69, srv1618541007=31, srv1676534302=36, srv90372414=95, srv843243797=89, srv215621063=63, srv961343931=97, srv1817424126=43, srv1484251986=24, srv656340696=81, srv110894546=6, srv989142683=99, srv114333048=8, srv1955155648=51, srv923684435=96, srv605857428=77, srv1417002278=22, srv598224544=76, srv547079489=73, srv1624086060=32, srv1303796180=18, srv1668521171=35, srv1866939687=45, srv1059643373=4, srv192458273=49, srv1612140197=30, srv40189880=68, srv961868291=98, srv1972617554=53, srv593687931=75, srv1059202006=3, srv736483397=86, srv342031718=66, srv869571398=91, srv149323629=25, srv864088695=90, srv1651510569=34, srv1891159215=46, srv70784319=84, srv1043523014=1, srv2143574828=62, srv656842104=82, srv204822900=55, srv1553176272=29, srv1529106056=28, srv882043855=92, srv1915231298=48, srv1094918969=5, srv1745944480=40, srv681538683=83, srv1742913167=39, srv2081510179=57, srv1512019402=26, srv1112984256=7, srv882892457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:53:02,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:53:02,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:53:02,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:53:02,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:53:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:53:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:53:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:53:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:53:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:53:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:53:02,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:53:02,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:53:02,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:53:02,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:53:02,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:53:02,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:53:02,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:53:02,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:53:02,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:53:02,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:53:02,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:53:02,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:53:02,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-27 03:53:02,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-27 03:53:02,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-27 03:53:02,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-27 03:53:02,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-27 03:53:02,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-27 03:53:02,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-27 03:53:02,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-27 03:53:02,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-27 03:53:02,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-27 03:53:02,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-27 03:53:02,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-27 03:53:02,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-27 03:53:02,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-27 03:53:02,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-27 03:53:02,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-27 03:53:02,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-27 03:53:02,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-27 03:53:02,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-27 03:53:02,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-27 03:53:02,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-27 03:53:02,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-27 03:53:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-27 03:53:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-27 03:53:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-27 03:53:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-27 03:53:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-27 03:53:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-27 03:53:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-27 03:53:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-27 03:53:02,719 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:02,720 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-27 03:53:02,720 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-27 03:53:03,078 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 674 ms to try 5000 different iterations.  Found a solution that moves 4669 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18076425529951465. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03857416338118914, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05229846143494474, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5187777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03857416338118914, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17566666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:03,101 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv591383002:81 , srv1043523014:82 , srv104926224:82 , srv1150043550:82 , srv1731829598:83 , srv2023963006:83 , srv342031718:83 , srv1059202006:84 , srv1218232710:84 , srv1369826944:84 , srv1618541007:84 , srv1724359191:84 , srv1891159215:84 , srv611908973:84 , srv70784319:84 , srv864088695:84 , srv1217467922:85 , srv1303796180:85 , srv1405668770:85 , srv1484251986:85 , srv149323629:85 , srv1529106056:85 , srv1612140197:85 , srv2055027646:85 , srv215621063:85 , srv605857428:85 , srv828372474:85 , srv90372414:85 , srv961868291:85 , srv1059643373:86 , srv1112984256:86 , srv114333048:86 , srv1153436788:86 , srv1198227000:86 , srv1265261864:86 , srv1417002278:86 , srv1512019402:86 , srv1624086060:86 , srv1668521171:86 , srv1745944480:86 , srv1817424126:86 , srv1896538800:86 , srv2081959520:86 , srv211402102:86 , srv2123130779:86 , srv2143574828:86 , srv40189880:86 , srv443439896:86 , srv463133369:86 , srv547079489:86 , srv681538683:86 , srv7272037:86 , srv882802978:86 , srv882892457:86 , srv1010890228:87 , srv1094918969:87 , srv1242254884:87 , srv1350920555:87 , srv1447531311:87 , srv1520984544:87 , srv1553176272:87 , srv1645822630:87 , srv1866939687:87 , srv1932310283:87 , srv1972325779:87 , srv2097235694:87 , srv598224544:87 , srv614294804:87 , srv633011247:87 , srv882043855:87 , srv1763474207:88 , srv1839830594:88 , srv1915231298:88 , srv192458273:88 , srv204822900:88 , srv393429428:88 , srv826093524:88 , srv961343931:88 , srv110894546:89 , srv1191087995:89 , srv1227928115:89 , srv1651510569:89 , srv1742913167:89 , srv1753567881:89 , srv1955155648:89 , srv23805227:89 , srv532694293:89 , srv656842104:89 , srv843243797:89 , srv869571398:89 , srv1676534302:90 , srv1972617554:90 , srv2081510179:90 , srv231522842:90 , srv736483397:90 , srv923684435:90 , srv593687931:91 , srv542257019:92 , srv656340696:93 , srv989142683:433 }
2023-09-27 03:53:03,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:53:03,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1150043550=9, srv1405668770=21, srv104926224=2, srv882802978=93, srv1932310283=50, srv2023963006=54, srv614294804=79, srv1731829598=38, srv1227928115=15, srv1217467922=13, srv2081959520=58, srv1753567881=41, srv542257019=72, srv1645822630=33, srv463133369=70, srv1265261864=17, srv1447531311=23, srv828372474=88, srv1191087995=11, srv2055027646=56, srv591383002=74, srv1010890228=0, srv7272037=85, srv1896538800=47, srv2097235694=59, srv23805227=65, srv826093524=87, srv1198227000=12, srv1972325779=52, srv633011247=80, srv1763474207=42, srv1242254884=16, srv1350920555=19, srv532694293=71, srv211402102=60, srv231522842=64, srv1520984544=27, srv1218232710=14, srv1724359191=37, srv1153436788=10, srv1839830594=44, srv1369826944=20, srv2123130779=61, srv393429428=67, srv611908973=78, srv443439896=69, srv1618541007=31, srv1676534302=36, srv90372414=95, srv843243797=89, srv215621063=63, srv961343931=97, srv1817424126=43, srv1484251986=24, srv656340696=81, srv110894546=6, srv989142683=99, srv114333048=8, srv1955155648=51, srv923684435=96, srv605857428=77, srv1417002278=22, srv598224544=76, srv547079489=73, srv1624086060=32, srv1303796180=18, srv1668521171=35, srv1866939687=45, srv1059643373=4, srv192458273=49, srv1612140197=30, srv40189880=68, srv961868291=98, srv1972617554=53, srv593687931=75, srv1059202006=3, srv736483397=86, srv342031718=66, srv869571398=91, srv149323629=25, srv864088695=90, srv1651510569=34, srv1891159215=46, srv70784319=84, srv1043523014=1, srv2143574828=62, srv656842104=82, srv204822900=55, srv1553176272=29, srv1529106056=28, srv882043855=92, srv1915231298=48, srv1094918969=5, srv1745944480=40, srv681538683=83, srv1742913167=39, srv2081510179=57, srv1512019402=26, srv1112984256=7, srv882892457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:53:03,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:53:03,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:53:03,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:53:03,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:53:03,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:53:03,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:53:03,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:53:03,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:53:03,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:53:03,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:53:03,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:53:03,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:53:03,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:53:03,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:53:03,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:53:03,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-27 03:53:03,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-27 03:53:03,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-27 03:53:03,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-27 03:53:03,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-27 03:53:03,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-27 03:53:03,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-27 03:53:03,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-27 03:53:03,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-27 03:53:03,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-27 03:53:03,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-27 03:53:03,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-27 03:53:03,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-27 03:53:03,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-27 03:53:03,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-27 03:53:03,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-27 03:53:03,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-27 03:53:03,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-27 03:53:03,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-27 03:53:03,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-27 03:53:03,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-27 03:53:03,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-27 03:53:03,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-27 03:53:03,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-27 03:53:03,376 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03857416338118914, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05229846143494474, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03857416338118914, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17566666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:03,376 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-27 03:53:03,376 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18073156226275236, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03857416338118914, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05229846143494474, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03857416338118914, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17566666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-27 03:53:03,707 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 467 ms to try 5000 different iterations.  Found a solution that moves 2667 regions; Going from a computed imbalance of 0.18073156226275236 to a new imbalance of 0.013969480193031458. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02216378399242671); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03217865393081416, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02216378399242671); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:03,739 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv656340696:79 , srv110894546:81 , srv1676534302:81 , srv1753567881:81 , srv1763474207:81 , srv923684435:81 , srv1972617554:82 , srv231522842:82 , srv532694293:82 , srv593687931:82 , srv114333048:83 , srv1198227000:83 , srv1227928115:83 , srv1242254884:83 , srv1520984544:83 , srv1839830594:83 , srv1866939687:83 , srv393429428:83 , srv542257019:83 , srv843243797:83 , srv869571398:83 , srv1010890228:84 , srv1191087995:84 , srv1265261864:84 , srv1417002278:84 , srv1645822630:84 , srv1955155648:84 , srv23805227:84 , srv547079489:84 , srv598224544:84 , srv656842104:84 , srv736483397:84 , srv961343931:84 , srv1094918969:85 , srv1218232710:85 , srv1405668770:85 , srv1447531311:85 , srv1512019402:85 , srv1529106056:85 , srv1553176272:85 , srv1624086060:85 , srv1742913167:85 , srv1817424126:85 , srv1972325779:85 , srv204822900:85 , srv2081510179:85 , srv826093524:85 , srv882043855:85 , srv882892457:85 , srv1112984256:86 , srv1217467922:86 , srv149323629:86 , srv1612140197:86 , srv1651510569:86 , srv1668521171:86 , srv1896538800:86 , srv192458273:86 , srv1932310283:86 , srv2081959520:86 , srv2097235694:86 , srv2123130779:86 , srv2143574828:86 , srv40189880:86 , srv443439896:86 , srv463133369:86 , srv614294804:86 , srv633011247:86 , srv7272037:86 , srv828372474:86 , srv882802978:86 , srv1059643373:87 , srv1618541007:87 , srv1724359191:87 , srv1745944480:87 , srv1915231298:87 , srv2055027646:87 , srv211402102:87 , srv681538683:87 , srv1059202006:88 , srv1153436788:88 , srv1303796180:88 , srv1350920555:88 , srv1369826944:88 , srv1484251986:88 , srv2023963006:88 , srv215621063:88 , srv605857428:88 , srv611908973:88 , srv90372414:88 , srv961868291:88 , srv104926224:89 , srv70784319:89 , srv864088695:89 , srv1043523014:90 , srv1731829598:90 , srv1891159215:90 , srv342031718:90 , srv1150043550:91 , srv591383002:93 , srv989142683:538 }
2023-09-27 03:53:03,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:53:03,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1150043550=9, srv1405668770=21, srv104926224=2, srv882802978=93, srv1932310283=50, srv2023963006=54, srv614294804=79, srv1731829598=38, srv1227928115=15, srv1217467922=13, srv2081959520=58, srv1753567881=41, srv542257019=72, srv1645822630=33, srv463133369=70, srv1265261864=17, srv1447531311=23, srv828372474=88, srv1191087995=11, srv2055027646=56, srv591383002=74, srv1010890228=0, srv7272037=85, srv1896538800=47, srv2097235694=59, srv23805227=65, srv826093524=87, srv1198227000=12, srv1972325779=52, srv633011247=80, srv1763474207=42, srv1242254884=16, srv1350920555=19, srv532694293=71, srv211402102=60, srv231522842=64, srv1520984544=27, srv1218232710=14, srv1724359191=37, srv1153436788=10, srv1839830594=44, srv1369826944=20, srv2123130779=61, srv393429428=67, srv611908973=78, srv443439896=69, srv1618541007=31, srv1676534302=36, srv90372414=95, srv843243797=89, srv215621063=63, srv961343931=97, srv1817424126=43, srv1484251986=24, srv656340696=81, srv110894546=6, srv989142683=99, srv114333048=8, srv1955155648=51, srv923684435=96, srv605857428=77, srv1417002278=22, srv598224544=76, srv547079489=73, srv1624086060=32, srv1303796180=18, srv1668521171=35, srv1866939687=45, srv1059643373=4, srv192458273=49, srv1612140197=30, srv40189880=68, srv961868291=98, srv1972617554=53, srv593687931=75, srv1059202006=3, srv736483397=86, srv342031718=66, srv869571398=91, srv149323629=25, srv864088695=90, srv1651510569=34, srv1891159215=46, srv70784319=84, srv1043523014=1, srv2143574828=62, srv656842104=82, srv204822900=55, srv1553176272=29, srv1529106056=28, srv882043855=92, srv1915231298=48, srv1094918969=5, srv1745944480=40, srv681538683=83, srv1742913167=39, srv2081510179=57, srv1512019402=26, srv1112984256=7, srv882892457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:53:03,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:53:03,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:53:03,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:53:03,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:53:03,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:53:03,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:53:03,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:53:03,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:53:03,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:53:03,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:53:03,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-27 03:53:03,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-27 03:53:03,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-27 03:53:03,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-27 03:53:03,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-27 03:53:03,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-27 03:53:03,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-27 03:53:03,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-27 03:53:03,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-27 03:53:03,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-27 03:53:03,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-27 03:53:03,960 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02216378399242671); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03217865393081416, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02216378399242671); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:03,961 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-27 03:53:03,961 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013950805459888384, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02216378399242671); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03217865393081416, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02216378399242671); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-27 03:53:04,251 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 403 ms to try 5000 different iterations.  Found a solution that moves 997 regions; Going from a computed imbalance of 0.013950805459888384 to a new imbalance of 7.967758498635781E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071137060360810895); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009511054469764316); MoveCostFunction : (multiplier=7.0, imbalance=0.11077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0071137060360810895); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:04,253 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1350920555:82 , srv1651510569:82 , srv1153436788:83 , srv1742913167:83 , srv1891159215:83 , srv2081510179:83 , srv1059643373:84 , srv1191087995:84 , srv1484251986:84 , srv1731829598:84 , srv192458273:84 , srv1932310283:84 , srv204822900:84 , srv211402102:84 , srv215621063:84 , srv23805227:84 , srv342031718:84 , srv443439896:84 , srv542257019:84 , srv591383002:84 , srv593687931:84 , srv605857428:84 , srv681538683:84 , srv70784319:84 , srv736483397:84 , srv90372414:84 , srv961868291:84 , srv1043523014:85 , srv1059202006:85 , srv1094918969:85 , srv1112984256:85 , srv1150043550:85 , srv1227928115:85 , srv1303796180:85 , srv1369826944:85 , srv1447531311:85 , srv1512019402:85 , srv1645822630:85 , srv1668521171:85 , srv1676534302:85 , srv1745944480:85 , srv1896538800:85 , srv1915231298:85 , srv1955155648:85 , srv1972325779:85 , srv2023963006:85 , srv2055027646:85 , srv2081959520:85 , srv2097235694:85 , srv2143574828:85 , srv231522842:85 , srv611908973:85 , srv614294804:85 , srv633011247:85 , srv656340696:85 , srv656842104:85 , srv826093524:85 , srv843243797:85 , srv864088695:85 , srv869571398:85 , srv882802978:85 , srv104926224:86 , srv1217467922:86 , srv1520984544:86 , srv1553176272:86 , srv1618541007:86 , srv1624086060:86 , srv1839830594:86 , srv1972617554:86 , srv2123130779:86 , srv393429428:86 , srv40189880:86 , srv463133369:86 , srv532694293:86 , srv598224544:86 , srv7272037:86 , srv961343931:86 , srv1010890228:87 , srv110894546:87 , srv1242254884:87 , srv1265261864:87 , srv1417002278:87 , srv149323629:87 , srv1612140197:87 , srv1724359191:87 , srv1753567881:87 , srv1817424126:87 , srv1866939687:87 , srv828372474:87 , srv882892457:87 , srv923684435:87 , srv1218232710:88 , srv1405668770:88 , srv1529106056:88 , srv1763474207:88 , srv547079489:88 , srv882043855:88 , srv114333048:89 , srv1198227000:89 , srv989142683:550 }
2023-09-27 03:53:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:53:04,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1150043550=9, srv1405668770=21, srv104926224=2, srv882802978=93, srv1932310283=50, srv2023963006=54, srv614294804=79, srv1731829598=38, srv1227928115=15, srv1217467922=13, srv2081959520=58, srv1753567881=41, srv542257019=72, srv1645822630=33, srv463133369=70, srv1265261864=17, srv1447531311=23, srv828372474=88, srv1191087995=11, srv2055027646=56, srv591383002=74, srv1010890228=0, srv7272037=85, srv1896538800=47, srv2097235694=59, srv23805227=65, srv826093524=87, srv1198227000=12, srv1972325779=52, srv633011247=80, srv1763474207=42, srv1242254884=16, srv1350920555=19, srv532694293=71, srv211402102=60, srv231522842=64, srv1520984544=27, srv1218232710=14, srv1724359191=37, srv1153436788=10, srv1839830594=44, srv1369826944=20, srv2123130779=61, srv393429428=67, srv611908973=78, srv443439896=69, srv1618541007=31, srv1676534302=36, srv90372414=95, srv843243797=89, srv215621063=63, srv961343931=97, srv1817424126=43, srv1484251986=24, srv656340696=81, srv110894546=6, srv989142683=99, srv114333048=8, srv1955155648=51, srv923684435=96, srv605857428=77, srv1417002278=22, srv598224544=76, srv547079489=73, srv1624086060=32, srv1303796180=18, srv1668521171=35, srv1866939687=45, srv1059643373=4, srv192458273=49, srv1612140197=30, srv40189880=68, srv961868291=98, srv1972617554=53, srv593687931=75, srv1059202006=3, srv736483397=86, srv342031718=66, srv869571398=91, srv149323629=25, srv864088695=90, srv1651510569=34, srv1891159215=46, srv70784319=84, srv1043523014=1, srv2143574828=62, srv656842104=82, srv204822900=55, srv1553176272=29, srv1529106056=28, srv882043855=92, srv1915231298=48, srv1094918969=5, srv1745944480=40, srv681538683=83, srv1742913167=39, srv2081510179=57, srv1512019402=26, srv1112984256=7, srv882892457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:53:04,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:53:04,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-27 03:53:04,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-27 03:53:04,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-27 03:53:04,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-27 03:53:04,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-27 03:53:04,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-27 03:53:04,366 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071137060360810895); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009511054469764316); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0071137060360810895); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:04,366 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-27 03:53:04,367 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.897947066526123E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071137060360810895); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009511054469764316); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0071137060360810895); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-27 03:53:04,689 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 402 ms to try 5000 different iterations.  Found a solution that moves 286 regions; Going from a computed imbalance of 7.897947066526123E-4 to a new imbalance of 2.007433493482792E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.03177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:04,691 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1915231298:82 , srv882043855:82 , srv542257019:83 , srv614294804:83 , srv1010890228:84 , srv1094918969:84 , srv114333048:84 , srv1150043550:84 , srv1198227000:84 , srv1303796180:84 , srv1405668770:84 , srv149323629:84 , srv1529106056:84 , srv1553176272:84 , srv1612140197:84 , srv1724359191:84 , srv1745944480:84 , srv1817424126:84 , srv192458273:84 , srv1955155648:84 , srv1972617554:84 , srv2081510179:84 , srv2097235694:84 , srv2123130779:84 , srv215621063:84 , srv40189880:84 , srv463133369:84 , srv547079489:84 , srv633011247:84 , srv656842104:84 , srv7272037:84 , srv736483397:84 , srv826093524:84 , srv828372474:84 , srv864088695:84 , srv882892457:84 , srv923684435:84 , srv961343931:84 , srv1043523014:85 , srv1059202006:85 , srv1059643373:85 , srv110894546:85 , srv1112984256:85 , srv1153436788:85 , srv1191087995:85 , srv1217467922:85 , srv1218232710:85 , srv1227928115:85 , srv1242254884:85 , srv1265261864:85 , srv1369826944:85 , srv1417002278:85 , srv1447531311:85 , srv1484251986:85 , srv1520984544:85 , srv1618541007:85 , srv1624086060:85 , srv1651510569:85 , srv1668521171:85 , srv1731829598:85 , srv1742913167:85 , srv1753567881:85 , srv1763474207:85 , srv1839830594:85 , srv1866939687:85 , srv1891159215:85 , srv1896538800:85 , srv1932310283:85 , srv1972325779:85 , srv204822900:85 , srv2055027646:85 , srv2081959520:85 , srv211402102:85 , srv2143574828:85 , srv231522842:85 , srv23805227:85 , srv393429428:85 , srv532694293:85 , srv591383002:85 , srv593687931:85 , srv598224544:85 , srv605857428:85 , srv611908973:85 , srv656340696:85 , srv681538683:85 , srv70784319:85 , srv843243797:85 , srv869571398:85 , srv882802978:85 , srv90372414:85 , srv961868291:85 , srv104926224:86 , srv1350920555:86 , srv1512019402:86 , srv1645822630:86 , srv1676534302:86 , srv2023963006:86 , srv342031718:86 , srv443439896:86 , srv989142683:621 }
2023-09-27 03:53:04,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:53:04,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1150043550=9, srv1405668770=21, srv104926224=2, srv882802978=93, srv1932310283=50, srv2023963006=54, srv614294804=79, srv1731829598=38, srv1227928115=15, srv1217467922=13, srv2081959520=58, srv1753567881=41, srv542257019=72, srv1645822630=33, srv463133369=70, srv1265261864=17, srv1447531311=23, srv828372474=88, srv1191087995=11, srv2055027646=56, srv591383002=74, srv1010890228=0, srv7272037=85, srv1896538800=47, srv2097235694=59, srv23805227=65, srv826093524=87, srv1198227000=12, srv1972325779=52, srv633011247=80, srv1763474207=42, srv1242254884=16, srv1350920555=19, srv532694293=71, srv211402102=60, srv231522842=64, srv1520984544=27, srv1218232710=14, srv1724359191=37, srv1153436788=10, srv1839830594=44, srv1369826944=20, srv2123130779=61, srv393429428=67, srv611908973=78, srv443439896=69, srv1618541007=31, srv1676534302=36, srv90372414=95, srv843243797=89, srv215621063=63, srv961343931=97, srv1817424126=43, srv1484251986=24, srv656340696=81, srv110894546=6, srv989142683=99, srv114333048=8, srv1955155648=51, srv923684435=96, srv605857428=77, srv1417002278=22, srv598224544=76, srv547079489=73, srv1624086060=32, srv1303796180=18, srv1668521171=35, srv1866939687=45, srv1059643373=4, srv192458273=49, srv1612140197=30, srv40189880=68, srv961868291=98, srv1972617554=53, srv593687931=75, srv1059202006=3, srv736483397=86, srv342031718=66, srv869571398=91, srv149323629=25, srv864088695=90, srv1651510569=34, srv1891159215=46, srv70784319=84, srv1043523014=1, srv2143574828=62, srv656842104=82, srv204822900=55, srv1553176272=29, srv1529106056=28, srv882043855=92, srv1915231298=48, srv1094918969=5, srv1745944480=40, srv681538683=83, srv1742913167=39, srv2081510179=57, srv1512019402=26, srv1112984256=7, srv882892457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:53:04,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:53:04,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:53:04,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-27 03:53:04,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-27 03:53:04,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-27 03:53:04,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-27 03:53:04,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-27 03:53:04,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-27 03:53:04,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-27 03:53:04,797 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8071720132083482E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:04,797 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1915231298:82 , srv882043855:82 , srv542257019:83 , srv614294804:83 , srv1010890228:84 , srv1094918969:84 , srv114333048:84 , srv1150043550:84 , srv1198227000:84 , srv1303796180:84 , srv1405668770:84 , srv149323629:84 , srv1529106056:84 , srv1553176272:84 , srv1612140197:84 , srv1724359191:84 , srv1745944480:84 , srv1817424126:84 , srv192458273:84 , srv1955155648:84 , srv1972617554:84 , srv2081510179:84 , srv2097235694:84 , srv2123130779:84 , srv215621063:84 , srv40189880:84 , srv463133369:84 , srv547079489:84 , srv633011247:84 , srv656842104:84 , srv7272037:84 , srv736483397:84 , srv826093524:84 , srv828372474:84 , srv864088695:84 , srv882892457:84 , srv923684435:84 , srv961343931:84 , srv1043523014:85 , srv1059202006:85 , srv1059643373:85 , srv110894546:85 , srv1112984256:85 , srv1153436788:85 , srv1191087995:85 , srv1217467922:85 , srv1218232710:85 , srv1227928115:85 , srv1242254884:85 , srv1265261864:85 , srv1369826944:85 , srv1417002278:85 , srv1447531311:85 , srv1484251986:85 , srv1520984544:85 , srv1618541007:85 , srv1624086060:85 , srv1651510569:85 , srv1668521171:85 , srv1731829598:85 , srv1742913167:85 , srv1753567881:85 , srv1763474207:85 , srv1839830594:85 , srv1866939687:85 , srv1891159215:85 , srv1896538800:85 , srv1932310283:85 , srv1972325779:85 , srv204822900:85 , srv2055027646:85 , srv2081959520:85 , srv211402102:85 , srv2143574828:85 , srv231522842:85 , srv23805227:85 , srv393429428:85 , srv532694293:85 , srv591383002:85 , srv593687931:85 , srv598224544:85 , srv605857428:85 , srv611908973:85 , srv656340696:85 , srv681538683:85 , srv70784319:85 , srv843243797:85 , srv869571398:85 , srv882802978:85 , srv90372414:85 , srv961868291:85 , srv104926224:86 , srv1350920555:86 , srv1512019402:86 , srv1645822630:86 , srv1676534302:86 , srv2023963006:86 , srv342031718:86 , srv443439896:86 , srv989142683:621 }
2023-09-27 03:53:04,824 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=621 (was 623), ProcessCount=180 (was 180), AvailableMemoryMB=2553 (was 3202)
2023-09-27 03:53:04,832 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=621, ProcessCount=180, AvailableMemoryMB=2553
2023-09-27 03:53:04,832 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-27 03:53:04,833 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-27 03:53:04,842 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1251551056:84 , srv1254573128:84 , srv1274169923:84 , srv1311878854:84 , srv1380097836:84 , srv1392665073:84 , srv1398968510:84 , srv1496021829:84 , srv1542945751:84 , srv1803013879:84 , srv1896764671:84 , srv189966317:84 , srv193838377:84 , srv1942048926:84 , srv1952900909:84 , srv2091346429:84 , srv274116357:84 , srv3307751:84 , srv343574028:84 , srv392169160:84 , srv514989154:84 , srv568704510:84 , srv580007197:84 , srv63858910:84 , srv666726324:84 , srv831297781:84 , srv836379122:84 , srv838164229:84 , srv87239751:84 , srv1553666222:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-27 03:53:04,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:53:04,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv392169160=20, srv87239751=29, srv568704510=22, srv666726324=25, srv836379122=27, srv1392665073=5, srv3307751=18, srv1398968510=6, srv1803013879=10, srv193838377=13, srv274116357=17, srv1274169923=2, srv1251551056=0, srv1942048926=14, srv1952900909=15, srv189966317=12, srv514989154=21, srv831297781=26, srv1496021829=7, srv838164229=28, srv2091346429=16, srv343574028=19, srv580007197=23, srv1542945751=8, srv1311878854=3, srv1553666222=9, srv1380097836=4, srv63858910=24, srv1254573128=1, srv1896764671=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:53:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:53:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:53:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:53:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:53:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:53:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:53:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:53:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:53:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:53:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:53:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:53:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-27 03:53:04,868 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2134688814038017, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:53:04,868 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912865676739181, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2134688814038017, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-27 03:55:04,857 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 8472837 different iterations.  Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912865676739181 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:55:04,864 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1251551056:90 , srv1254573128:90 , srv1274169923:90 , srv1311878854:90 , srv1380097836:90 , srv1392665073:90 , srv1398968510:90 , srv1496021829:90 , srv1542945751:90 , srv1553666222:90 , srv1803013879:90 , srv1896764671:90 , srv189966317:90 , srv193838377:90 , srv1942048926:90 , srv1952900909:90 , srv2091346429:90 , srv274116357:90 , srv3307751:90 , srv343574028:90 , srv392169160:90 , srv514989154:90 , srv568704510:90 , srv580007197:90 , srv63858910:90 , srv666726324:90 , srv831297781:90 , srv836379122:90 , srv838164229:90 , srv87239751:90 }
2023-09-27 03:55:04,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-27 03:55:04,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv392169160=20, srv87239751=29, srv568704510=22, srv666726324=25, srv836379122=27, srv1392665073=5, srv3307751=18, srv1398968510=6, srv1803013879=10, srv193838377=13, srv274116357=17, srv1274169923=2, srv1251551056=0, srv1942048926=14, srv1952900909=15, srv189966317=12, srv514989154=21, srv831297781=26, srv1496021829=7, srv838164229=28, srv2091346429=16, srv343574028=19, srv580007197=23, srv1542945751=8, srv1311878854=3, srv1553666222=9, srv1380097836=4, srv63858910=24, srv1254573128=1, srv1896764671=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-27 03:55:04,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-27 03:55:04,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-27 03:55:04,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-27 03:55:04,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-27 03:55:04,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-27 03:55:04,883 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-27 03:55:04,884 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1251551056:90 , srv1254573128:90 , srv1274169923:90 , srv1311878854:90 , srv1380097836:90 , srv1392665073:90 , srv1398968510:90 , srv1496021829:90 , srv1542945751:90 , srv1553666222:90 , srv1803013879:90 , srv1896764671:90 , srv189966317:90 , srv193838377:90 , srv1942048926:90 , srv1952900909:90 , srv2091346429:90 , srv274116357:90 , srv3307751:90 , srv343574028:90 , srv392169160:90 , srv514989154:90 , srv568704510:90 , srv580007197:90 , srv63858910:90 , srv666726324:90 , srv831297781:90 , srv836379122:90 , srv838164229:90 , srv87239751:90 }
2023-09-27 03:55:04,895 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=396 (was 621), ProcessCount=180 (was 180), AvailableMemoryMB=1438 (was 2553)