2024-03-28 06:53:27,796 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-28 06:53:28,091 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-28 06:53:28,193 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-28 06:53:28,255 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-28 06:53:28,262 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-28 06:53:28,277 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=171, AvailableMemoryMB=10767
2024-03-28 06:53:28,279 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-28 06:53:28,280 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-28 06:53:28,387 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016376012:84 , srv1054859643:84 , srv1073260133:84 , srv1084798994:84 , srv1098482409:84 , srv1132315302:84 , srv1151063362:84 , srv1151260969:84 , srv1185333033:84 , srv1194122704:84 , srv1201451347:84 , srv1204984663:84 , srv1227862055:84 , srv1263922497:84 , srv126500503:84 , srv1276179921:84 , srv1310990370:84 , srv142454461:84 , srv1425433132:84 , srv1438749841:84 , srv1469069560:84 , srv1474891257:84 , srv1482049946:84 , srv1488223501:84 , srv1492222326:84 , srv150079902:84 , srv1509993715:84 , srv1517019192:84 , srv1525605759:84 , srv1580773774:84 , srv1592802516:84 , srv159879206:84 , srv1635826416:84 , srv164746147:84 , srv166285866:84 , srv1666341233:84 , srv1674989943:84 , srv1709468871:84 , srv1710012124:84 , srv1718916250:84 , srv1744732973:84 , srv1752312244:84 , srv175258828:84 , srv1757973451:84 , srv1798039764:84 , srv1806503779:84 , srv1806602595:84 , srv1856464501:84 , srv1858990854:84 , srv1929461828:84 , srv1950113360:84 , srv1970102146:84 , srv1986007839:84 , srv2020725539:84 , srv202194397:84 , srv2037799779:84 , srv2065095002:84 , srv2074976027:84 , srv2084871699:84 , srv2089028897:84 , srv2093394183:84 , srv2110424646:84 , srv2113119565:84 , srv221223502:84 , srv269922601:84 , srv288391848:84 , srv297365254:84 , srv326984032:84 , srv331824457:84 , srv357038638:84 , srv394802814:84 , srv468503417:84 , srv500220591:84 , srv527000372:84 , srv539120615:84 , srv565432283:84 , srv618176993:84 , srv627491565:84 , srv628935261:84 , srv67145922:84 , srv679641448:84 , srv693451679:84 , srv6997349:84 , srv700014826:84 , srv705286361:84 , srv727463157:84 , srv734494826:84 , srv759704854:84 , srv763322366:84 , srv795389933:84 , srv865528557:84 , srv87115334:84 , srv893901183:84 , srv912300593:84 , srv914848939:84 , srv937899784:84 , srv939519018:84 , srv949826399:84 , srv972649097:84 , srv501483899:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-28 06:53:28,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:53:28,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2089028897=59, srv1276179921=15, srv1425433132=18, srv1482049946=22, srv500220591=72, srv126500503=14, srv1185333033=8, srv297365254=66, srv1718916250=39, srv1806503779=45, srv893901183=93, srv1929461828=49, srv700014826=84, srv1151063362=6, srv2113119565=62, srv1098482409=4, srv1950113360=50, srv693451679=82, srv221223502=63, srv1806602595=46, srv6997349=83, srv159879206=31, srv2074976027=57, srv914848939=95, srv1194122704=9, srv939519018=97, srv1474891257=21, srv734494826=87, srv202194397=54, srv87115334=92, srv1517019192=27, srv1635826416=32, srv1592802516=30, srv1151260969=7, srv142454461=17, srv1438749841=19, srv949826399=98, srv1488223501=23, srv331824457=68, srv1710012124=38, srv1856464501=47, srv628935261=79, srv1201451347=10, srv1858990854=48, srv618176993=77, srv1016376012=0, srv1709468871=37, srv1798039764=44, srv1073260133=2, srv150079902=25, srv539120615=75, srv1084798994=3, srv705286361=85, srv1580773774=29, srv1744732973=40, srv937899784=96, srv175258828=42, srv527000372=74, srv2093394183=60, srv1204984663=11, srv2065095002=56, srv1263922497=13, srv1970102146=51, srv394802814=70, srv679641448=81, srv795389933=90, srv164746147=33, srv1310990370=16, srv565432283=76, srv357038638=69, srv1492222326=24, srv468503417=71, srv1469069560=20, srv912300593=94, srv1054859643=1, srv2084871699=58, srv2037799779=55, srv972649097=99, srv1986007839=52, srv326984032=67, srv1757973451=43, srv269922601=64, srv1674989943=36, srv288391848=65, srv763322366=89, srv1752312244=41, srv1666341233=35, srv727463157=86, srv1509993715=26, srv627491565=78, srv1132315302=5, srv1227862055=12, srv166285866=34, srv759704854=88, srv865528557=91, srv501483899=73, srv2110424646=61, srv67145922=80, srv2020725539=53, srv1525605759=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-28 06:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-28 06:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-28 06:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-28 06:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-28 06:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-28 06:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-28 06:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-28 06:53:28,633 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:28,634 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-28 06:53:28,634 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-28 06:53:29,043 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 537 ms to try 5000 different iterations.  Found a solution that moves 4647 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18293991855794142. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0378381375591727, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051452526799265944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5163333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0378381375591727, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17808333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:29,060 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1744732973:81 , srv1151260969:83 , srv1227862055:83 , srv1310990370:83 , srv1425433132:83 , srv1488223501:83 , srv1517019192:83 , srv1709468871:83 , srv175258828:83 , srv705286361:83 , srv1084798994:84 , srv1482049946:84 , srv1525605759:84 , srv1580773774:84 , srv1710012124:84 , srv1798039764:84 , srv2093394183:84 , srv627491565:84 , srv628935261:84 , srv693451679:84 , srv87115334:84 , srv939519018:84 , srv1054859643:85 , srv1204984663:85 , srv1263922497:85 , srv126500503:85 , srv1276179921:85 , srv1509993715:85 , srv1635826416:85 , srv1666341233:85 , srv2084871699:85 , srv221223502:85 , srv269922601:85 , srv288391848:85 , srv357038638:85 , srv394802814:85 , srv527000372:85 , srv618176993:85 , srv727463157:85 , srv734494826:85 , srv763322366:85 , srv1016376012:86 , srv1185333033:86 , srv1194122704:86 , srv1201451347:86 , srv1492222326:86 , srv1806503779:86 , srv1950113360:86 , srv2037799779:86 , srv2065095002:86 , srv2074976027:86 , srv2110424646:86 , srv326984032:86 , srv539120615:86 , srv565432283:86 , srv937899784:86 , srv1073260133:87 , srv1151063362:87 , srv142454461:87 , srv1438749841:87 , srv1718916250:87 , srv1858990854:87 , srv1970102146:87 , srv1986007839:87 , srv2113119565:87 , srv6997349:87 , srv700014826:87 , srv893901183:87 , srv1098482409:88 , srv1132315302:88 , srv1469069560:88 , srv166285866:88 , srv1674989943:88 , srv331824457:88 , srv865528557:88 , srv912300593:88 , srv914848939:88 , srv1474891257:89 , srv1592802516:89 , srv297365254:89 , srv67145922:89 , srv795389933:89 , srv949826399:89 , srv159879206:90 , srv1752312244:90 , srv1806602595:90 , srv202194397:90 , srv500220591:90 , srv759704854:90 , srv150079902:91 , srv164746147:91 , srv1757973451:91 , srv1856464501:91 , srv468503417:91 , srv2089028897:92 , srv972649097:92 , srv2020725539:93 , srv679641448:93 , srv1929461828:98 , srv501483899:426 }
2024-03-28 06:53:29,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:53:29,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2089028897=59, srv1276179921=15, srv1425433132=18, srv1482049946=22, srv500220591=72, srv126500503=14, srv1185333033=8, srv297365254=66, srv1718916250=39, srv1806503779=45, srv893901183=93, srv1929461828=49, srv700014826=84, srv1151063362=6, srv2113119565=62, srv1098482409=4, srv1950113360=50, srv693451679=82, srv221223502=63, srv1806602595=46, srv6997349=83, srv159879206=31, srv2074976027=57, srv914848939=95, srv1194122704=9, srv939519018=97, srv1474891257=21, srv734494826=87, srv202194397=54, srv87115334=92, srv1517019192=27, srv1635826416=32, srv1592802516=30, srv1151260969=7, srv142454461=17, srv1438749841=19, srv949826399=98, srv1488223501=23, srv331824457=68, srv1710012124=38, srv1856464501=47, srv628935261=79, srv1201451347=10, srv1858990854=48, srv618176993=77, srv1016376012=0, srv1709468871=37, srv1798039764=44, srv1073260133=2, srv150079902=25, srv539120615=75, srv1084798994=3, srv705286361=85, srv1580773774=29, srv1744732973=40, srv937899784=96, srv175258828=42, srv527000372=74, srv2093394183=60, srv1204984663=11, srv2065095002=56, srv1263922497=13, srv1970102146=51, srv394802814=70, srv679641448=81, srv795389933=90, srv164746147=33, srv1310990370=16, srv565432283=76, srv357038638=69, srv1492222326=24, srv468503417=71, srv1469069560=20, srv912300593=94, srv1054859643=1, srv2084871699=58, srv2037799779=55, srv972649097=99, srv1986007839=52, srv326984032=67, srv1757973451=43, srv269922601=64, srv1674989943=36, srv288391848=65, srv763322366=89, srv1752312244=41, srv1666341233=35, srv727463157=86, srv1509993715=26, srv627491565=78, srv1132315302=5, srv1227862055=12, srv166285866=34, srv759704854=88, srv865528557=91, srv501483899=73, srv2110424646=61, srv67145922=80, srv2020725539=53, srv1525605759=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:53:29,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:53:29,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:53:29,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:53:29,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:53:29,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:53:29,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:53:29,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:53:29,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:53:29,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:53:29,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:53:29,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:53:29,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:53:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-28 06:53:29,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-28 06:53:29,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-28 06:53:29,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-28 06:53:29,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-28 06:53:29,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-28 06:53:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-28 06:53:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-28 06:53:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-28 06:53:29,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-28 06:53:29,263 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0378381375591727, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051452526799265944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0378381375591727, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17808333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:29,264 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-28 06:53:29,264 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18290737956847167, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0378381375591727, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051452526799265944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0378381375591727, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17808333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-28 06:53:29,514 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 381 ms to try 5000 different iterations.  Found a solution that moves 2713 regions; Going from a computed imbalance of 0.18290737956847167 to a new imbalance of 0.016951175292271407. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02016331870933543); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03330975597138436, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30144444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02016331870933543); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:29,524 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1929461828:76 , srv679641448:79 , srv150079902:80 , srv164746147:80 , srv2020725539:80 , srv2089028897:80 , srv468503417:80 , srv1752312244:81 , srv1856464501:81 , srv500220591:81 , srv1592802516:82 , srv1757973451:82 , srv202194397:82 , srv759704854:82 , srv795389933:82 , srv972649097:82 , srv1438749841:83 , srv166285866:83 , srv1718916250:83 , srv1806602595:83 , srv331824457:83 , srv865528557:83 , srv893901183:83 , srv914848939:83 , srv949826399:83 , srv1016376012:84 , srv1132315302:84 , srv1151063362:84 , srv1204984663:84 , srv1474891257:84 , srv912300593:84 , srv1073260133:85 , srv1194122704:85 , srv1201451347:85 , srv142454461:85 , srv1492222326:85 , srv1806503779:85 , srv2084871699:85 , srv2113119565:85 , srv221223502:85 , srv288391848:85 , srv297365254:85 , srv539120615:85 , srv565432283:85 , srv67145922:85 , srv6997349:85 , srv1098482409:86 , srv1469069560:86 , srv1482049946:86 , srv1509993715:86 , srv1674989943:86 , srv1858990854:86 , srv1950113360:86 , srv1970102146:86 , srv1986007839:86 , srv2037799779:86 , srv2110424646:86 , srv326984032:86 , srv527000372:86 , srv618176993:86 , srv734494826:86 , srv1084798994:87 , srv1185333033:87 , srv1580773774:87 , srv159879206:87 , srv1635826416:87 , srv1666341233:87 , srv2065095002:87 , srv357038638:87 , srv693451679:87 , srv700014826:87 , srv763322366:87 , srv939519018:87 , srv1054859643:88 , srv1263922497:88 , srv126500503:88 , srv1425433132:88 , srv1488223501:88 , srv1525605759:88 , srv269922601:88 , srv627491565:88 , srv628935261:88 , srv705286361:88 , srv727463157:88 , srv1276179921:89 , srv1710012124:89 , srv1744732973:89 , srv937899784:89 , srv1517019192:90 , srv2074976027:90 , srv2093394183:90 , srv87115334:90 , srv1151260969:91 , srv1227862055:91 , srv1310990370:91 , srv1709468871:91 , srv394802814:91 , srv175258828:92 , srv1798039764:92 , srv501483899:527 }
2024-03-28 06:53:29,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:53:29,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2089028897=59, srv1276179921=15, srv1425433132=18, srv1482049946=22, srv500220591=72, srv126500503=14, srv1185333033=8, srv297365254=66, srv1718916250=39, srv1806503779=45, srv893901183=93, srv1929461828=49, srv700014826=84, srv1151063362=6, srv2113119565=62, srv1098482409=4, srv1950113360=50, srv693451679=82, srv221223502=63, srv1806602595=46, srv6997349=83, srv159879206=31, srv2074976027=57, srv914848939=95, srv1194122704=9, srv939519018=97, srv1474891257=21, srv734494826=87, srv202194397=54, srv87115334=92, srv1517019192=27, srv1635826416=32, srv1592802516=30, srv1151260969=7, srv142454461=17, srv1438749841=19, srv949826399=98, srv1488223501=23, srv331824457=68, srv1710012124=38, srv1856464501=47, srv628935261=79, srv1201451347=10, srv1858990854=48, srv618176993=77, srv1016376012=0, srv1709468871=37, srv1798039764=44, srv1073260133=2, srv150079902=25, srv539120615=75, srv1084798994=3, srv705286361=85, srv1580773774=29, srv1744732973=40, srv937899784=96, srv175258828=42, srv527000372=74, srv2093394183=60, srv1204984663=11, srv2065095002=56, srv1263922497=13, srv1970102146=51, srv394802814=70, srv679641448=81, srv795389933=90, srv164746147=33, srv1310990370=16, srv565432283=76, srv357038638=69, srv1492222326=24, srv468503417=71, srv1469069560=20, srv912300593=94, srv1054859643=1, srv2084871699=58, srv2037799779=55, srv972649097=99, srv1986007839=52, srv326984032=67, srv1757973451=43, srv269922601=64, srv1674989943=36, srv288391848=65, srv763322366=89, srv1752312244=41, srv1666341233=35, srv727463157=86, srv1509993715=26, srv627491565=78, srv1132315302=5, srv1227862055=12, srv166285866=34, srv759704854=88, srv865528557=91, srv501483899=73, srv2110424646=61, srv67145922=80, srv2020725539=53, srv1525605759=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:53:29,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:53:29,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-28 06:53:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-28 06:53:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-28 06:53:29,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-28 06:53:29,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-28 06:53:29,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-28 06:53:29,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-28 06:53:29,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-28 06:53:29,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-28 06:53:29,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-28 06:53:29,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-28 06:53:29,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-28 06:53:29,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-28 06:53:29,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-28 06:53:29,682 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02016331870933543); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03330975597138436, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02016331870933543); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:29,682 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-28 06:53:29,682 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016932178460243974, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02016331870933543); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03330975597138436, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02016331870933543); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-28 06:53:29,917 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 334 ms to try 5000 different iterations.  Found a solution that moves 989 regions; Going from a computed imbalance of 0.016932178460243974 to a new imbalance of 5.584641375572504E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006716981845877292); MoveCostFunction : (multiplier=7.0, imbalance=0.10988888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:29,919 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv159879206:81 , srv2074976027:82 , srv937899784:82 , srv1227862055:83 , srv1276179921:83 , srv1310990370:83 , srv1469069560:83 , srv175258828:83 , srv1798039764:83 , srv1929461828:83 , srv2093394183:83 , srv394802814:83 , srv67145922:83 , srv700014826:83 , srv1054859643:84 , srv1098482409:84 , srv1151260969:84 , srv1185333033:84 , srv1263922497:84 , srv1517019192:84 , srv1709468871:84 , srv1757973451:84 , srv1858990854:84 , srv1970102146:84 , srv2020725539:84 , srv269922601:84 , srv297365254:84 , srv727463157:84 , srv87115334:84 , srv1073260133:85 , srv1084798994:85 , srv1132315302:85 , srv126500503:85 , srv142454461:85 , srv1474891257:85 , srv1525605759:85 , srv1666341233:85 , srv1674989943:85 , srv1710012124:85 , srv1806602595:85 , srv1986007839:85 , srv202194397:85 , srv2065095002:85 , srv326984032:85 , srv357038638:85 , srv627491565:85 , srv6997349:85 , srv759704854:85 , srv763322366:85 , srv972649097:85 , srv1194122704:86 , srv1492222326:86 , srv150079902:86 , srv1509993715:86 , srv1580773774:86 , srv1592802516:86 , srv1635826416:86 , srv166285866:86 , srv1806503779:86 , srv1856464501:86 , srv1950113360:86 , srv2037799779:86 , srv2089028897:86 , srv2110424646:86 , srv331824457:86 , srv539120615:86 , srv565432283:86 , srv628935261:86 , srv679641448:86 , srv693451679:86 , srv705286361:86 , srv734494826:86 , srv912300593:86 , srv949826399:86 , srv1016376012:87 , srv1151063362:87 , srv1201451347:87 , srv1425433132:87 , srv1488223501:87 , srv164746147:87 , srv1744732973:87 , srv1752312244:87 , srv2084871699:87 , srv2113119565:87 , srv221223502:87 , srv468503417:87 , srv500220591:87 , srv527000372:87 , srv618176993:87 , srv795389933:87 , srv865528557:87 , srv893901183:87 , srv914848939:87 , srv939519018:87 , srv1204984663:88 , srv1438749841:88 , srv1482049946:88 , srv1718916250:88 , srv288391848:88 , srv501483899:553 }
2024-03-28 06:53:29,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:53:29,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2089028897=59, srv1276179921=15, srv1425433132=18, srv1482049946=22, srv500220591=72, srv126500503=14, srv1185333033=8, srv297365254=66, srv1718916250=39, srv1806503779=45, srv893901183=93, srv1929461828=49, srv700014826=84, srv1151063362=6, srv2113119565=62, srv1098482409=4, srv1950113360=50, srv693451679=82, srv221223502=63, srv1806602595=46, srv6997349=83, srv159879206=31, srv2074976027=57, srv914848939=95, srv1194122704=9, srv939519018=97, srv1474891257=21, srv734494826=87, srv202194397=54, srv87115334=92, srv1517019192=27, srv1635826416=32, srv1592802516=30, srv1151260969=7, srv142454461=17, srv1438749841=19, srv949826399=98, srv1488223501=23, srv331824457=68, srv1710012124=38, srv1856464501=47, srv628935261=79, srv1201451347=10, srv1858990854=48, srv618176993=77, srv1016376012=0, srv1709468871=37, srv1798039764=44, srv1073260133=2, srv150079902=25, srv539120615=75, srv1084798994=3, srv705286361=85, srv1580773774=29, srv1744732973=40, srv937899784=96, srv175258828=42, srv527000372=74, srv2093394183=60, srv1204984663=11, srv2065095002=56, srv1263922497=13, srv1970102146=51, srv394802814=70, srv679641448=81, srv795389933=90, srv164746147=33, srv1310990370=16, srv565432283=76, srv357038638=69, srv1492222326=24, srv468503417=71, srv1469069560=20, srv912300593=94, srv1054859643=1, srv2084871699=58, srv2037799779=55, srv972649097=99, srv1986007839=52, srv326984032=67, srv1757973451=43, srv269922601=64, srv1674989943=36, srv288391848=65, srv763322366=89, srv1752312244=41, srv1666341233=35, srv727463157=86, srv1509993715=26, srv627491565=78, srv1132315302=5, srv1227862055=12, srv166285866=34, srv759704854=88, srv865528557=91, srv501483899=73, srv2110424646=61, srv67145922=80, srv2020725539=53, srv1525605759=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:53:29,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:53:29,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:53:29,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:53:29,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:53:29,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:53:29,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:53:29,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-28 06:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-28 06:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-28 06:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-28 06:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-28 06:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-28 06:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-28 06:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-28 06:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-28 06:53:30,069 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006716981845877292); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:30,069 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-28 06:53:30,070 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.515390115435642E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006716981845877292); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-28 06:53:30,414 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 451 ms to try 5000 different iterations.  Found a solution that moves 246 regions; Going from a computed imbalance of 5.515390115435642E-4 to a new imbalance of 1.1565421855410234E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.027333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:30,415 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1674989943:83 , srv1752312244:83 , srv1798039764:83 , srv1806602595:83 , srv2113119565:83 , srv394802814:83 , srv972649097:83 , srv1054859643:84 , srv1098482409:84 , srv1151063362:84 , srv1151260969:84 , srv1185333033:84 , srv1201451347:84 , srv1263922497:84 , srv126500503:84 , srv1425433132:84 , srv1438749841:84 , srv1474891257:84 , srv1482049946:84 , srv1488223501:84 , srv159879206:84 , srv1635826416:84 , srv164746147:84 , srv1718916250:84 , srv175258828:84 , srv1856464501:84 , srv1950113360:84 , srv1986007839:84 , srv2037799779:84 , srv2065095002:84 , srv2074976027:84 , srv2089028897:84 , srv2110424646:84 , srv288391848:84 , srv297365254:84 , srv468503417:84 , srv500220591:84 , srv527000372:84 , srv618176993:84 , srv628935261:84 , srv679641448:84 , srv795389933:84 , srv87115334:84 , srv893901183:84 , srv912300593:84 , srv914848939:84 , srv939519018:84 , srv949826399:84 , srv1073260133:85 , srv1132315302:85 , srv1194122704:85 , srv1204984663:85 , srv1227862055:85 , srv1276179921:85 , srv1310990370:85 , srv142454461:85 , srv1469069560:85 , srv1492222326:85 , srv150079902:85 , srv1509993715:85 , srv1517019192:85 , srv1525605759:85 , srv1580773774:85 , srv166285866:85 , srv1666341233:85 , srv1709468871:85 , srv1710012124:85 , srv1744732973:85 , srv1757973451:85 , srv1806503779:85 , srv1858990854:85 , srv1929461828:85 , srv1970102146:85 , srv2020725539:85 , srv202194397:85 , srv2084871699:85 , srv2093394183:85 , srv221223502:85 , srv269922601:85 , srv326984032:85 , srv331824457:85 , srv357038638:85 , srv539120615:85 , srv565432283:85 , srv67145922:85 , srv693451679:85 , srv6997349:85 , srv700014826:85 , srv705286361:85 , srv727463157:85 , srv734494826:85 , srv759704854:85 , srv763322366:85 , srv865528557:85 , srv937899784:85 , srv1016376012:86 , srv1084798994:86 , srv1592802516:86 , srv627491565:86 , srv501483899:636 }
2024-03-28 06:53:30,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:53:30,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2089028897=59, srv1276179921=15, srv1425433132=18, srv1482049946=22, srv500220591=72, srv126500503=14, srv1185333033=8, srv297365254=66, srv1718916250=39, srv1806503779=45, srv893901183=93, srv1929461828=49, srv700014826=84, srv1151063362=6, srv2113119565=62, srv1098482409=4, srv1950113360=50, srv693451679=82, srv221223502=63, srv1806602595=46, srv6997349=83, srv159879206=31, srv2074976027=57, srv914848939=95, srv1194122704=9, srv939519018=97, srv1474891257=21, srv734494826=87, srv202194397=54, srv87115334=92, srv1517019192=27, srv1635826416=32, srv1592802516=30, srv1151260969=7, srv142454461=17, srv1438749841=19, srv949826399=98, srv1488223501=23, srv331824457=68, srv1710012124=38, srv1856464501=47, srv628935261=79, srv1201451347=10, srv1858990854=48, srv618176993=77, srv1016376012=0, srv1709468871=37, srv1798039764=44, srv1073260133=2, srv150079902=25, srv539120615=75, srv1084798994=3, srv705286361=85, srv1580773774=29, srv1744732973=40, srv937899784=96, srv175258828=42, srv527000372=74, srv2093394183=60, srv1204984663=11, srv2065095002=56, srv1263922497=13, srv1970102146=51, srv394802814=70, srv679641448=81, srv795389933=90, srv164746147=33, srv1310990370=16, srv565432283=76, srv357038638=69, srv1492222326=24, srv468503417=71, srv1469069560=20, srv912300593=94, srv1054859643=1, srv2084871699=58, srv2037799779=55, srv972649097=99, srv1986007839=52, srv326984032=67, srv1757973451=43, srv269922601=64, srv1674989943=36, srv288391848=65, srv763322366=89, srv1752312244=41, srv1666341233=35, srv727463157=86, srv1509993715=26, srv627491565=78, srv1132315302=5, srv1227862055=12, srv166285866=34, srv759704854=88, srv865528557=91, srv501483899=73, srv2110424646=61, srv67145922=80, srv2020725539=53, srv1525605759=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:53:30,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:53:30,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:53:30,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:53:30,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-28 06:53:30,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-28 06:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-28 06:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-28 06:53:30,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-28 06:53:30,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-28 06:53:30,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-28 06:53:30,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-28 06:53:30,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-28 06:53:30,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-28 06:53:30,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-28 06:53:30,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-28 06:53:30,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-28 06:53:30,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-28 06:53:30,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-28 06:53:30,562 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.842893039063616E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:30,562 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1674989943:83 , srv1752312244:83 , srv1798039764:83 , srv1806602595:83 , srv2113119565:83 , srv394802814:83 , srv972649097:83 , srv1054859643:84 , srv1098482409:84 , srv1151063362:84 , srv1151260969:84 , srv1185333033:84 , srv1201451347:84 , srv1263922497:84 , srv126500503:84 , srv1425433132:84 , srv1438749841:84 , srv1474891257:84 , srv1482049946:84 , srv1488223501:84 , srv159879206:84 , srv1635826416:84 , srv164746147:84 , srv1718916250:84 , srv175258828:84 , srv1856464501:84 , srv1950113360:84 , srv1986007839:84 , srv2037799779:84 , srv2065095002:84 , srv2074976027:84 , srv2089028897:84 , srv2110424646:84 , srv288391848:84 , srv297365254:84 , srv468503417:84 , srv500220591:84 , srv527000372:84 , srv618176993:84 , srv628935261:84 , srv679641448:84 , srv795389933:84 , srv87115334:84 , srv893901183:84 , srv912300593:84 , srv914848939:84 , srv939519018:84 , srv949826399:84 , srv1073260133:85 , srv1132315302:85 , srv1194122704:85 , srv1204984663:85 , srv1227862055:85 , srv1276179921:85 , srv1310990370:85 , srv142454461:85 , srv1469069560:85 , srv1492222326:85 , srv150079902:85 , srv1509993715:85 , srv1517019192:85 , srv1525605759:85 , srv1580773774:85 , srv166285866:85 , srv1666341233:85 , srv1709468871:85 , srv1710012124:85 , srv1744732973:85 , srv1757973451:85 , srv1806503779:85 , srv1858990854:85 , srv1929461828:85 , srv1970102146:85 , srv2020725539:85 , srv202194397:85 , srv2084871699:85 , srv2093394183:85 , srv221223502:85 , srv269922601:85 , srv326984032:85 , srv331824457:85 , srv357038638:85 , srv539120615:85 , srv565432283:85 , srv67145922:85 , srv693451679:85 , srv6997349:85 , srv700014826:85 , srv705286361:85 , srv727463157:85 , srv734494826:85 , srv759704854:85 , srv763322366:85 , srv865528557:85 , srv937899784:85 , srv1016376012:86 , srv1084798994:86 , srv1592802516:86 , srv627491565:86 , srv501483899:636 }
2024-03-28 06:53:30,590 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 242) - SystemLoadAverage LEAK? -, ProcessCount=177 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=10292 (was 10767)
2024-03-28 06:53:30,600 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=177, AvailableMemoryMB=10291
2024-03-28 06:53:30,601 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-28 06:53:30,601 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-28 06:53:30,613 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026574968:84 , srv1060623061:84 , srv1073976272:84 , srv1099796575:84 , srv1125533848:84 , srv11700026:84 , srv1207644272:84 , srv1221790598:84 , srv1246401064:84 , srv1284633083:84 , srv1332394946:84 , srv1451619996:84 , srv1496546697:84 , srv1556407575:84 , srv1706523225:84 , srv170758354:84 , srv1717299515:84 , srv1785296046:84 , srv189176988:84 , srv1911316914:84 , srv1931615771:84 , srv478624941:84 , srv583030062:84 , srv654884777:84 , srv658288473:84 , srv718332677:84 , srv792876246:84 , srv878751591:84 , srv960467283:84 , srv256743735:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-28 06:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv960467283=29, srv1785296046=17, srv1496546697=12, srv256743735=21, srv11700026=5, srv1284633083=9, srv1060623061=1, srv658288473=25, srv792876246=27, srv1332394946=10, srv878751591=28, srv1073976272=2, srv1717299515=16, srv583030062=23, srv1706523225=14, srv1451619996=11, srv478624941=22, srv718332677=26, srv189176988=18, srv1246401064=8, srv1026574968=0, srv1125533848=4, srv170758354=15, srv1911316914=19, srv654884777=24, srv1207644272=6, srv1556407575=13, srv1221790598=7, srv1099796575=3, srv1931615771=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:53:30,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:53:30,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-28 06:53:30,643 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1334623665176984, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:53:30,643 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991261357882323, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1334623665176984, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-28 06:55:30,628 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5721325 different iterations.  Found a solution that moves 1863 regions; Going from a computed imbalance of 0.991261357882323 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:55:30,633 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1026574968:90 , srv1060623061:90 , srv1073976272:90 , srv1099796575:90 , srv1125533848:90 , srv11700026:90 , srv1207644272:90 , srv1221790598:90 , srv1246401064:90 , srv1284633083:90 , srv1332394946:90 , srv1451619996:90 , srv1496546697:90 , srv1556407575:90 , srv1706523225:90 , srv170758354:90 , srv1717299515:90 , srv1785296046:90 , srv189176988:90 , srv1911316914:90 , srv1931615771:90 , srv256743735:90 , srv478624941:90 , srv583030062:90 , srv654884777:90 , srv658288473:90 , srv718332677:90 , srv792876246:90 , srv878751591:90 , srv960467283:90 }
2024-03-28 06:55:30,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-28 06:55:30,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv960467283=29, srv1785296046=17, srv1496546697=12, srv256743735=21, srv11700026=5, srv1284633083=9, srv1060623061=1, srv658288473=25, srv792876246=27, srv1332394946=10, srv878751591=28, srv1073976272=2, srv1717299515=16, srv583030062=23, srv1706523225=14, srv1451619996=11, srv478624941=22, srv718332677=26, srv189176988=18, srv1246401064=8, srv1026574968=0, srv1125533848=4, srv170758354=15, srv1911316914=19, srv654884777=24, srv1207644272=6, srv1556407575=13, srv1221790598=7, srv1099796575=3, srv1931615771=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-28 06:55:30,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-28 06:55:30,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-28 06:55:30,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-28 06:55:30,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-28 06:55:30,651 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-28 06:55:30,651 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1026574968:90 , srv1060623061:90 , srv1073976272:90 , srv1099796575:90 , srv1125533848:90 , srv11700026:90 , srv1207644272:90 , srv1221790598:90 , srv1246401064:90 , srv1284633083:90 , srv1332394946:90 , srv1451619996:90 , srv1496546697:90 , srv1556407575:90 , srv1706523225:90 , srv170758354:90 , srv1717299515:90 , srv1785296046:90 , srv189176988:90 , srv1911316914:90 , srv1931615771:90 , srv256743735:90 , srv478624941:90 , srv583030062:90 , srv654884777:90 , srv658288473:90 , srv718332677:90 , srv792876246:90 , srv878751591:90 , srv960467283:90 }
2024-03-28 06:55:30,665 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=160 (was 254), ProcessCount=168 (was 177), AvailableMemoryMB=9516 (was 10291)