2024-03-27 02:53:29,678 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-27 02:53:29,961 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-27 02:53:30,077 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-27 02:53:30,138 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-27 02:53:30,145 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-27 02:53:30,160 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=171, AvailableMemoryMB=11647
2024-03-27 02:53:30,162 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-27 02:53:30,162 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-27 02:53:30,285 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015484063:84 , srv104650934:84 , srv1057990159:84 , srv111353409:84 , srv1122391184:84 , srv1123208112:84 , srv1158561781:84 , srv1158820400:84 , srv1158827886:84 , srv1175786253:84 , srv1181788090:84 , srv1187497526:84 , srv1203054426:84 , srv1225497734:84 , srv1227714985:84 , srv1243166146:84 , srv1266249026:84 , srv1273749751:84 , srv1353499766:84 , srv1376774874:84 , srv1385826061:84 , srv1397700708:84 , srv1404044131:84 , srv1410395138:84 , srv1495149957:84 , srv1497641951:84 , srv1501645516:84 , srv1519251071:84 , srv1555029688:84 , srv1617503395:84 , srv1635528994:84 , srv1665106732:84 , srv1691218122:84 , srv170908819:84 , srv1715251222:84 , srv1764358294:84 , srv1778944419:84 , srv1782155539:84 , srv1786428402:84 , srv1801268403:84 , srv1860412617:84 , srv1885366308:84 , srv1889333257:84 , srv1904885117:84 , srv1938306206:84 , srv1968774305:84 , srv1986669091:84 , srv2036579240:84 , srv204843230:84 , srv206526864:84 , srv2069912803:84 , srv208877973:84 , srv2126818258:84 , srv2134986586:84 , srv224746465:84 , srv226467261:84 , srv260995828:84 , srv274070405:84 , srv315901980:84 , srv356268106:84 , srv363424070:84 , srv418076945:84 , srv431140409:84 , srv451313626:84 , srv456558335:84 , srv506153249:84 , srv520069801:84 , srv527871553:84 , srv528946170:84 , srv581649051:84 , srv584515758:84 , srv588008089:84 , srv600500887:84 , srv606708126:84 , srv622229273:84 , srv624527331:84 , srv636674055:84 , srv664136480:84 , srv678974722:84 , srv699388060:84 , srv704458253:84 , srv712564342:84 , srv732392409:84 , srv739247110:84 , srv754181708:84 , srv777752805:84 , srv780179330:84 , srv809897909:84 , srv83848181:84 , srv8407032:84 , srv859967834:84 , srv872679622:84 , srv882679582:84 , srv885162946:84 , srv885515251:84 , srv929275006:84 , srv934154131:84 , srv961928172:84 , srv978312531:84 , srv826422264:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-27 02:53:30,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:53:30,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv636674055=76, srv2036579240=47, srv664136480=77, srv1617503395=29, srv1938306206=44, srv882679582=93, srv111353409=3, srv872679622=92, srv581649051=69, srv315901980=58, srv1273749751=17, srv451313626=63, srv712564342=81, srv1227714985=14, srv885162946=94, srv1501645516=26, srv1691218122=32, srv1057990159=2, srv527871553=67, srv456558335=64, srv1519251071=27, srv732392409=82, srv104650934=1, srv600500887=72, srv1665106732=31, srv1904885117=43, srv431140409=62, srv204843230=48, srv206526864=49, srv777752805=85, srv1353499766=18, srv1158820400=7, srv2134986586=53, srv1715251222=34, srv622229273=74, srv224746465=54, srv739247110=83, srv699388060=79, srv1495149957=24, srv1860412617=40, srv780179330=86, srv885515251=95, srv1885366308=41, srv678974722=78, srv528946170=68, srv1497641951=25, srv1555029688=28, srv826422264=88, srv1158561781=6, srv934154131=97, srv1410395138=23, srv226467261=55, srv1123208112=5, srv1187497526=11, srv1635528994=30, srv2069912803=50, srv754181708=84, srv520069801=66, srv584515758=70, srv260995828=56, srv1889333257=42, srv1243166146=15, srv961928172=98, srv1122391184=4, srv929275006=96, srv606708126=73, srv1203054426=12, srv1986669091=46, srv356268106=59, srv1158827886=8, srv624527331=75, srv83848181=89, srv208877973=51, srv1376774874=19, srv1015484063=0, srv1968774305=45, srv978312531=99, srv859967834=91, srv1404044131=22, srv8407032=90, srv1225497734=13, srv704458253=80, srv274070405=57, srv1266249026=16, srv170908819=33, srv1778944419=36, srv1786428402=38, srv1397700708=21, srv1181788090=10, srv1764358294=35, srv588008089=71, srv809897909=87, srv2126818258=52, srv418076945=61, srv1175786253=9, srv1385826061=20, srv1801268403=39, srv1782155539=37, srv363424070=60, srv506153249=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:53:30,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 02:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 02:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 02:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 02:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 02:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 02:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 02:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 02:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 02:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 02:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 02:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 02:53:30,598 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:30,598 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 02:53:30,598 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 02:53:31,011 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 610 ms to try 5000 different iterations.  Found a solution that moves 4704 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1776714964495223. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891725757006985, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053141489980534896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5226666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891725757006985, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:31,021 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1203054426:82 , srv363424070:82 , srv1181788090:83 , srv1397700708:83 , srv1665106732:83 , srv274070405:83 , srv431140409:83 , srv581649051:83 , srv961928172:83 , srv1158561781:84 , srv1187497526:84 , srv1617503395:84 , srv1691218122:84 , srv170908819:84 , srv204843230:84 , srv2069912803:84 , srv224746465:84 , srv678974722:84 , srv712564342:84 , srv8407032:84 , srv104650934:85 , srv111353409:85 , srv1122391184:85 , srv1385826061:85 , srv1519251071:85 , srv1715251222:85 , srv1782155539:85 , srv1938306206:85 , srv206526864:85 , srv226467261:85 , srv260995828:85 , srv527871553:85 , srv600500887:85 , srv882679582:85 , srv978312531:85 , srv1015484063:86 , srv1225497734:86 , srv1227714985:86 , srv1266249026:86 , srv1353499766:86 , srv1404044131:86 , srv1410395138:86 , srv1497641951:86 , srv1764358294:86 , srv1904885117:86 , srv1968774305:86 , srv1986669091:86 , srv622229273:86 , srv664136480:86 , srv699388060:86 , srv732392409:86 , srv777752805:86 , srv1376774874:87 , srv1501645516:87 , srv1778944419:87 , srv1801268403:87 , srv1860412617:87 , srv1889333257:87 , srv208877973:87 , srv2134986586:87 , srv418076945:87 , srv606708126:87 , srv624527331:87 , srv754181708:87 , srv780179330:87 , srv83848181:87 , srv872679622:87 , srv934154131:87 , srv1057990159:88 , srv1158827886:88 , srv1495149957:88 , srv1635528994:88 , srv1885366308:88 , srv2126818258:88 , srv356268106:88 , srv451313626:88 , srv584515758:88 , srv588008089:88 , srv636674055:88 , srv704458253:88 , srv859967834:88 , srv929275006:88 , srv1158820400:89 , srv1243166146:89 , srv2036579240:89 , srv1123208112:90 , srv1273749751:90 , srv1555029688:90 , srv1786428402:90 , srv315901980:90 , srv456558335:90 , srv520069801:90 , srv739247110:90 , srv809897909:90 , srv885162946:90 , srv1175786253:91 , srv506153249:91 , srv528946170:91 , srv885515251:94 , srv826422264:436 }
2024-03-27 02:53:31,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:53:31,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv636674055=76, srv2036579240=47, srv664136480=77, srv1617503395=29, srv1938306206=44, srv882679582=93, srv111353409=3, srv872679622=92, srv581649051=69, srv315901980=58, srv1273749751=17, srv451313626=63, srv712564342=81, srv1227714985=14, srv885162946=94, srv1501645516=26, srv1691218122=32, srv1057990159=2, srv527871553=67, srv456558335=64, srv1519251071=27, srv732392409=82, srv104650934=1, srv600500887=72, srv1665106732=31, srv1904885117=43, srv431140409=62, srv204843230=48, srv206526864=49, srv777752805=85, srv1353499766=18, srv1158820400=7, srv2134986586=53, srv1715251222=34, srv622229273=74, srv224746465=54, srv739247110=83, srv699388060=79, srv1495149957=24, srv1860412617=40, srv780179330=86, srv885515251=95, srv1885366308=41, srv678974722=78, srv528946170=68, srv1497641951=25, srv1555029688=28, srv826422264=88, srv1158561781=6, srv934154131=97, srv1410395138=23, srv226467261=55, srv1123208112=5, srv1187497526=11, srv1635528994=30, srv2069912803=50, srv754181708=84, srv520069801=66, srv584515758=70, srv260995828=56, srv1889333257=42, srv1243166146=15, srv961928172=98, srv1122391184=4, srv929275006=96, srv606708126=73, srv1203054426=12, srv1986669091=46, srv356268106=59, srv1158827886=8, srv624527331=75, srv83848181=89, srv208877973=51, srv1376774874=19, srv1015484063=0, srv1968774305=45, srv978312531=99, srv859967834=91, srv1404044131=22, srv8407032=90, srv1225497734=13, srv704458253=80, srv274070405=57, srv1266249026=16, srv170908819=33, srv1778944419=36, srv1786428402=38, srv1397700708=21, srv1181788090=10, srv1764358294=35, srv588008089=71, srv809897909=87, srv2126818258=52, srv418076945=61, srv1175786253=9, srv1385826061=20, srv1801268403=39, srv1782155539=37, srv363424070=60, srv506153249=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:53:31,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:53:31,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:53:31,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 02:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 02:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 02:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 02:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 02:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 02:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 02:53:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 02:53:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 02:53:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 02:53:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 02:53:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 02:53:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 02:53:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 02:53:31,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891725757006985, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053141489980534896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891725757006985, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:31,196 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 02:53:31,196 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1776385583375219, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891725757006985, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053141489980534896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891725757006985, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 02:53:31,475 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 403 ms to try 5000 different iterations.  Found a solution that moves 2638 regions; Going from a computed imbalance of 0.1776385583375219 to a new imbalance of 0.014862709495720345. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020044843105623662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625882763368457, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2931111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020044843105623662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:31,485 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv885515251:78 , srv1273749751:80 , srv528946170:80 , srv885162946:80 , srv1123208112:81 , srv1175786253:81 , srv1786428402:81 , srv2036579240:81 , srv520069801:81 , srv739247110:81 , srv506153249:82 , srv809897909:82 , srv1555029688:83 , srv315901980:83 , srv356268106:83 , srv456558335:83 , srv859967834:83 , srv104650934:84 , srv1158820400:84 , srv1635528994:84 , srv1885366308:84 , srv2126818258:84 , srv2134986586:84 , srv451313626:84 , srv588008089:84 , srv872679622:84 , srv929275006:84 , srv978312531:84 , srv1015484063:85 , srv111353409:85 , srv1266249026:85 , srv1410395138:85 , srv1495149957:85 , srv1501645516:85 , srv1764358294:85 , srv1778944419:85 , srv1860412617:85 , srv1889333257:85 , srv1968774305:85 , srv260995828:85 , srv584515758:85 , srv606708126:85 , srv636674055:85 , srv732392409:85 , srv780179330:85 , srv83848181:85 , srv1122391184:86 , srv1158827886:86 , srv1187497526:86 , srv1243166146:86 , srv1385826061:86 , srv1519251071:86 , srv1938306206:86 , srv206526864:86 , srv208877973:86 , srv224746465:86 , srv418076945:86 , srv527871553:86 , srv581649051:86 , srv622229273:86 , srv624527331:86 , srv699388060:86 , srv704458253:86 , srv754181708:86 , srv882679582:86 , srv934154131:86 , srv1057990159:87 , srv1227714985:87 , srv1376774874:87 , srv170908819:87 , srv1801268403:87 , srv1986669091:87 , srv226467261:87 , srv1158561781:88 , srv1353499766:88 , srv1497641951:88 , srv1715251222:88 , srv431140409:88 , srv664136480:88 , srv678974722:88 , srv712564342:88 , srv1181788090:89 , srv1225497734:89 , srv1404044131:89 , srv1617503395:89 , srv1665106732:89 , srv1904885117:89 , srv204843230:89 , srv274070405:89 , srv600500887:89 , srv777752805:89 , srv8407032:89 , srv1782155539:90 , srv2069912803:90 , srv363424070:90 , srv1203054426:91 , srv1691218122:91 , srv961928172:91 , srv1397700708:92 , srv826422264:516 }
2024-03-27 02:53:31,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:53:31,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv636674055=76, srv2036579240=47, srv664136480=77, srv1617503395=29, srv1938306206=44, srv882679582=93, srv111353409=3, srv872679622=92, srv581649051=69, srv315901980=58, srv1273749751=17, srv451313626=63, srv712564342=81, srv1227714985=14, srv885162946=94, srv1501645516=26, srv1691218122=32, srv1057990159=2, srv527871553=67, srv456558335=64, srv1519251071=27, srv732392409=82, srv104650934=1, srv600500887=72, srv1665106732=31, srv1904885117=43, srv431140409=62, srv204843230=48, srv206526864=49, srv777752805=85, srv1353499766=18, srv1158820400=7, srv2134986586=53, srv1715251222=34, srv622229273=74, srv224746465=54, srv739247110=83, srv699388060=79, srv1495149957=24, srv1860412617=40, srv780179330=86, srv885515251=95, srv1885366308=41, srv678974722=78, srv528946170=68, srv1497641951=25, srv1555029688=28, srv826422264=88, srv1158561781=6, srv934154131=97, srv1410395138=23, srv226467261=55, srv1123208112=5, srv1187497526=11, srv1635528994=30, srv2069912803=50, srv754181708=84, srv520069801=66, srv584515758=70, srv260995828=56, srv1889333257=42, srv1243166146=15, srv961928172=98, srv1122391184=4, srv929275006=96, srv606708126=73, srv1203054426=12, srv1986669091=46, srv356268106=59, srv1158827886=8, srv624527331=75, srv83848181=89, srv208877973=51, srv1376774874=19, srv1015484063=0, srv1968774305=45, srv978312531=99, srv859967834=91, srv1404044131=22, srv8407032=90, srv1225497734=13, srv704458253=80, srv274070405=57, srv1266249026=16, srv170908819=33, srv1778944419=36, srv1786428402=38, srv1397700708=21, srv1181788090=10, srv1764358294=35, srv588008089=71, srv809897909=87, srv2126818258=52, srv418076945=61, srv1175786253=9, srv1385826061=20, srv1801268403=39, srv1782155539=37, srv363424070=60, srv506153249=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:53:31,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:53:31,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:53:31,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:53:31,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 02:53:31,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 02:53:31,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 02:53:31,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 02:53:31,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 02:53:31,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 02:53:31,654 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020044843105623662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625882763368457, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020044843105623662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:31,654 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 02:53:31,654 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014844237824917408, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020044843105623662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625882763368457, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020044843105623662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 02:53:31,903 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 368 ms to try 5000 different iterations.  Found a solution that moves 1043 regions; Going from a computed imbalance of 0.014844237824917408 to a new imbalance of 5.963420431605512E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006453764691102064); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008951797592365465); MoveCostFunction : (multiplier=7.0, imbalance=0.11588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006453764691102064); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:31,905 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1057990159:82 , srv1691218122:82 , srv1158820400:83 , srv1158827886:83 , srv1225497734:83 , srv1243166146:83 , srv1353499766:83 , srv1397700708:83 , srv1404044131:83 , srv1782155539:83 , srv1904885117:83 , srv600500887:83 , srv664136480:83 , srv961928172:83 , srv1181788090:84 , srv1203054426:84 , srv1227714985:84 , srv1376774874:84 , srv1495149957:84 , srv1497641951:84 , srv1665106732:84 , srv1715251222:84 , srv1801268403:84 , srv2069912803:84 , srv315901980:84 , srv418076945:84 , srv451313626:84 , srv624527331:84 , srv704458253:84 , srv754181708:84 , srv777752805:84 , srv8407032:84 , srv934154131:84 , srv1122391184:85 , srv1410395138:85 , srv1519251071:85 , srv1555029688:85 , srv1617503395:85 , srv1778944419:85 , srv1860412617:85 , srv1885366308:85 , srv1889333257:85 , srv1986669091:85 , srv204843230:85 , srv208877973:85 , srv274070405:85 , srv363424070:85 , srv431140409:85 , srv456558335:85 , srv506153249:85 , srv584515758:85 , srv588008089:85 , srv636674055:85 , srv678974722:85 , srv699388060:85 , srv809897909:85 , srv83848181:85 , srv885515251:85 , srv929275006:85 , srv1158561781:86 , srv1175786253:86 , srv1266249026:86 , srv1385826061:86 , srv1635528994:86 , srv1938306206:86 , srv1968774305:86 , srv206526864:86 , srv2126818258:86 , srv226467261:86 , srv260995828:86 , srv356268106:86 , srv606708126:86 , srv712564342:86 , srv732392409:86 , srv739247110:86 , srv780179330:86 , srv859967834:86 , srv1015484063:87 , srv1123208112:87 , srv1187497526:87 , srv1273749751:87 , srv1501645516:87 , srv1764358294:87 , srv1786428402:87 , srv2036579240:87 , srv2134986586:87 , srv224746465:87 , srv520069801:87 , srv527871553:87 , srv528946170:87 , srv872679622:87 , srv882679582:87 , srv885162946:87 , srv111353409:88 , srv622229273:88 , srv978312531:88 , srv104650934:89 , srv170908819:89 , srv581649051:89 , srv826422264:563 }
2024-03-27 02:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv636674055=76, srv2036579240=47, srv664136480=77, srv1617503395=29, srv1938306206=44, srv882679582=93, srv111353409=3, srv872679622=92, srv581649051=69, srv315901980=58, srv1273749751=17, srv451313626=63, srv712564342=81, srv1227714985=14, srv885162946=94, srv1501645516=26, srv1691218122=32, srv1057990159=2, srv527871553=67, srv456558335=64, srv1519251071=27, srv732392409=82, srv104650934=1, srv600500887=72, srv1665106732=31, srv1904885117=43, srv431140409=62, srv204843230=48, srv206526864=49, srv777752805=85, srv1353499766=18, srv1158820400=7, srv2134986586=53, srv1715251222=34, srv622229273=74, srv224746465=54, srv739247110=83, srv699388060=79, srv1495149957=24, srv1860412617=40, srv780179330=86, srv885515251=95, srv1885366308=41, srv678974722=78, srv528946170=68, srv1497641951=25, srv1555029688=28, srv826422264=88, srv1158561781=6, srv934154131=97, srv1410395138=23, srv226467261=55, srv1123208112=5, srv1187497526=11, srv1635528994=30, srv2069912803=50, srv754181708=84, srv520069801=66, srv584515758=70, srv260995828=56, srv1889333257=42, srv1243166146=15, srv961928172=98, srv1122391184=4, srv929275006=96, srv606708126=73, srv1203054426=12, srv1986669091=46, srv356268106=59, srv1158827886=8, srv624527331=75, srv83848181=89, srv208877973=51, srv1376774874=19, srv1015484063=0, srv1968774305=45, srv978312531=99, srv859967834=91, srv1404044131=22, srv8407032=90, srv1225497734=13, srv704458253=80, srv274070405=57, srv1266249026=16, srv170908819=33, srv1778944419=36, srv1786428402=38, srv1397700708=21, srv1181788090=10, srv1764358294=35, srv588008089=71, srv809897909=87, srv2126818258=52, srv418076945=61, srv1175786253=9, srv1385826061=20, srv1801268403=39, srv1782155539=37, srv363424070=60, srv506153249=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:53:31,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:53:31,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:53:31,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 02:53:31,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 02:53:31,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 02:53:31,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 02:53:31,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 02:53:31,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 02:53:31,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 02:53:31,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 02:53:31,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 02:53:32,076 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006453764691102064); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008951797592365465); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006453764691102064); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:32,076 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 02:53:32,077 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.890388010652281E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006453764691102064); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008951797592365465); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006453764691102064); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 02:53:32,445 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations.  Found a solution that moves 210 regions; Going from a computed imbalance of 5.890388010652281E-4 to a new imbalance of 1.0695132142087808E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.023333333333333334); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:32,446 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv170908819:82 , srv622229273:82 , srv1015484063:83 , srv1501645516:83 , srv1555029688:83 , srv704458253:83 , srv777752805:83 , srv104650934:84 , srv111353409:84 , srv1123208112:84 , srv1158561781:84 , srv1158827886:84 , srv1175786253:84 , srv1225497734:84 , srv1243166146:84 , srv1266249026:84 , srv1353499766:84 , srv1376774874:84 , srv1397700708:84 , srv1404044131:84 , srv1497641951:84 , srv1617503395:84 , srv1635528994:84 , srv1764358294:84 , srv1778944419:84 , srv1782155539:84 , srv1786428402:84 , srv1801268403:84 , srv1904885117:84 , srv204843230:84 , srv2126818258:84 , srv2134986586:84 , srv226467261:84 , srv456558335:84 , srv506153249:84 , srv520069801:84 , srv527871553:84 , srv528946170:84 , srv581649051:84 , srv584515758:84 , srv606708126:84 , srv636674055:84 , srv712564342:84 , srv780179330:84 , srv872679622:84 , srv1057990159:85 , srv1187497526:85 , srv1203054426:85 , srv1273749751:85 , srv1385826061:85 , srv1495149957:85 , srv1519251071:85 , srv1691218122:85 , srv1860412617:85 , srv1885366308:85 , srv1889333257:85 , srv1938306206:85 , srv1968774305:85 , srv1986669091:85 , srv2036579240:85 , srv206526864:85 , srv2069912803:85 , srv208877973:85 , srv224746465:85 , srv274070405:85 , srv315901980:85 , srv356268106:85 , srv363424070:85 , srv418076945:85 , srv588008089:85 , srv600500887:85 , srv624527331:85 , srv664136480:85 , srv678974722:85 , srv699388060:85 , srv732392409:85 , srv739247110:85 , srv754181708:85 , srv809897909:85 , srv83848181:85 , srv8407032:85 , srv859967834:85 , srv882679582:85 , srv885515251:85 , srv929275006:85 , srv934154131:85 , srv961928172:85 , srv978312531:85 , srv1122391184:86 , srv1158820400:86 , srv1181788090:86 , srv1227714985:86 , srv1665106732:86 , srv1715251222:86 , srv260995828:86 , srv431140409:86 , srv451313626:86 , srv885162946:86 , srv1410395138:87 , srv826422264:627 }
2024-03-27 02:53:32,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:53:32,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv636674055=76, srv2036579240=47, srv664136480=77, srv1617503395=29, srv1938306206=44, srv882679582=93, srv111353409=3, srv872679622=92, srv581649051=69, srv315901980=58, srv1273749751=17, srv451313626=63, srv712564342=81, srv1227714985=14, srv885162946=94, srv1501645516=26, srv1691218122=32, srv1057990159=2, srv527871553=67, srv456558335=64, srv1519251071=27, srv732392409=82, srv104650934=1, srv600500887=72, srv1665106732=31, srv1904885117=43, srv431140409=62, srv204843230=48, srv206526864=49, srv777752805=85, srv1353499766=18, srv1158820400=7, srv2134986586=53, srv1715251222=34, srv622229273=74, srv224746465=54, srv739247110=83, srv699388060=79, srv1495149957=24, srv1860412617=40, srv780179330=86, srv885515251=95, srv1885366308=41, srv678974722=78, srv528946170=68, srv1497641951=25, srv1555029688=28, srv826422264=88, srv1158561781=6, srv934154131=97, srv1410395138=23, srv226467261=55, srv1123208112=5, srv1187497526=11, srv1635528994=30, srv2069912803=50, srv754181708=84, srv520069801=66, srv584515758=70, srv260995828=56, srv1889333257=42, srv1243166146=15, srv961928172=98, srv1122391184=4, srv929275006=96, srv606708126=73, srv1203054426=12, srv1986669091=46, srv356268106=59, srv1158827886=8, srv624527331=75, srv83848181=89, srv208877973=51, srv1376774874=19, srv1015484063=0, srv1968774305=45, srv978312531=99, srv859967834=91, srv1404044131=22, srv8407032=90, srv1225497734=13, srv704458253=80, srv274070405=57, srv1266249026=16, srv170908819=33, srv1778944419=36, srv1786428402=38, srv1397700708=21, srv1181788090=10, srv1764358294=35, srv588008089=71, srv809897909=87, srv2126818258=52, srv418076945=61, srv1175786253=9, srv1385826061=20, srv1801268403=39, srv1782155539=37, srv363424070=60, srv506153249=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:53:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:53:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 02:53:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 02:53:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 02:53:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 02:53:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 02:53:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 02:53:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 02:53:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 02:53:32,572 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.224680713499232E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:32,573 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv170908819:82 , srv622229273:82 , srv1015484063:83 , srv1501645516:83 , srv1555029688:83 , srv704458253:83 , srv777752805:83 , srv104650934:84 , srv111353409:84 , srv1123208112:84 , srv1158561781:84 , srv1158827886:84 , srv1175786253:84 , srv1225497734:84 , srv1243166146:84 , srv1266249026:84 , srv1353499766:84 , srv1376774874:84 , srv1397700708:84 , srv1404044131:84 , srv1497641951:84 , srv1617503395:84 , srv1635528994:84 , srv1764358294:84 , srv1778944419:84 , srv1782155539:84 , srv1786428402:84 , srv1801268403:84 , srv1904885117:84 , srv204843230:84 , srv2126818258:84 , srv2134986586:84 , srv226467261:84 , srv456558335:84 , srv506153249:84 , srv520069801:84 , srv527871553:84 , srv528946170:84 , srv581649051:84 , srv584515758:84 , srv606708126:84 , srv636674055:84 , srv712564342:84 , srv780179330:84 , srv872679622:84 , srv1057990159:85 , srv1187497526:85 , srv1203054426:85 , srv1273749751:85 , srv1385826061:85 , srv1495149957:85 , srv1519251071:85 , srv1691218122:85 , srv1860412617:85 , srv1885366308:85 , srv1889333257:85 , srv1938306206:85 , srv1968774305:85 , srv1986669091:85 , srv2036579240:85 , srv206526864:85 , srv2069912803:85 , srv208877973:85 , srv224746465:85 , srv274070405:85 , srv315901980:85 , srv356268106:85 , srv363424070:85 , srv418076945:85 , srv588008089:85 , srv600500887:85 , srv624527331:85 , srv664136480:85 , srv678974722:85 , srv699388060:85 , srv732392409:85 , srv739247110:85 , srv754181708:85 , srv809897909:85 , srv83848181:85 , srv8407032:85 , srv859967834:85 , srv882679582:85 , srv885515251:85 , srv929275006:85 , srv934154131:85 , srv961928172:85 , srv978312531:85 , srv1122391184:86 , srv1158820400:86 , srv1181788090:86 , srv1227714985:86 , srv1665106732:86 , srv1715251222:86 , srv260995828:86 , srv431140409:86 , srv451313626:86 , srv885162946:86 , srv1410395138:87 , srv826422264:627 }
2024-03-27 02:53:32,598 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=285 (was 285), ProcessCount=171 (was 171), AvailableMemoryMB=11156 (was 11647)
2024-03-27 02:53:32,608 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=171, AvailableMemoryMB=11156
2024-03-27 02:53:32,609 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-27 02:53:32,609 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-27 02:53:32,621 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1088103147:84 , srv1096539871:84 , srv1294838976:84 , srv1329874623:84 , srv135501637:84 , srv1464294062:84 , srv1664655979:84 , srv1692630481:84 , srv1727727807:84 , srv1731208600:84 , srv1731395359:84 , srv1794069546:84 , srv1820517095:84 , srv1854716228:84 , srv1916289172:84 , srv1955527026:84 , srv1983208332:84 , srv250823579:84 , srv330941525:84 , srv536816740:84 , srv558841718:84 , srv564363121:84 , srv57146088:84 , srv608129354:84 , srv764642647:84 , srv795344614:84 , srv815142393:84 , srv827982248:84 , srv894558573:84 , srv218997982:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-27 02:53:32,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:53:32,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv536816740=20, srv815142393=27, srv827982248=28, srv1664655979=6, srv57146088=23, srv1731208600=9, srv1096539871=1, srv564363121=22, srv1794069546=11, srv330941525=19, srv1329874623=3, srv894558573=29, srv558841718=21, srv1294838976=2, srv1916289172=14, srv1820517095=12, srv1854716228=13, srv135501637=4, srv1692630481=7, srv218997982=17, srv1088103147=0, srv1727727807=8, srv1983208332=16, srv764642647=25, srv1731395359=10, srv1955527026=15, srv250823579=18, srv608129354=24, srv795344614=26, srv1464294062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:53:32,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:53:32,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:53:32,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-27 02:53:32,658 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1075841558199535, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:53:32,659 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912532037426323, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1075841558199535, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-27 02:55:32,642 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5477680 different iterations.  Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912532037426323 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:55:32,648 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1088103147:90 , srv1096539871:90 , srv1294838976:90 , srv1329874623:90 , srv135501637:90 , srv1464294062:90 , srv1664655979:90 , srv1692630481:90 , srv1727727807:90 , srv1731208600:90 , srv1731395359:90 , srv1794069546:90 , srv1820517095:90 , srv1854716228:90 , srv1916289172:90 , srv1955527026:90 , srv1983208332:90 , srv218997982:90 , srv250823579:90 , srv330941525:90 , srv536816740:90 , srv558841718:90 , srv564363121:90 , srv57146088:90 , srv608129354:90 , srv764642647:90 , srv795344614:90 , srv815142393:90 , srv827982248:90 , srv894558573:90 }
2024-03-27 02:55:32,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 02:55:32,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv536816740=20, srv815142393=27, srv827982248=28, srv1664655979=6, srv57146088=23, srv1731208600=9, srv1096539871=1, srv564363121=22, srv1794069546=11, srv330941525=19, srv1329874623=3, srv894558573=29, srv558841718=21, srv1294838976=2, srv1916289172=14, srv1820517095=12, srv1854716228=13, srv135501637=4, srv1692630481=7, srv218997982=17, srv1088103147=0, srv1727727807=8, srv1983208332=16, srv764642647=25, srv1731395359=10, srv1955527026=15, srv250823579=18, srv608129354=24, srv795344614=26, srv1464294062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 02:55:32,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 02:55:32,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 02:55:32,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 02:55:32,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-27 02:55:32,666 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 02:55:32,667 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1088103147:90 , srv1096539871:90 , srv1294838976:90 , srv1329874623:90 , srv135501637:90 , srv1464294062:90 , srv1664655979:90 , srv1692630481:90 , srv1727727807:90 , srv1731208600:90 , srv1731395359:90 , srv1794069546:90 , srv1820517095:90 , srv1854716228:90 , srv1916289172:90 , srv1955527026:90 , srv1983208332:90 , srv218997982:90 , srv250823579:90 , srv330941525:90 , srv536816740:90 , srv558841718:90 , srv564363121:90 , srv57146088:90 , srv608129354:90 , srv764642647:90 , srv795344614:90 , srv815142393:90 , srv827982248:90 , srv894558573:90 }
2024-03-27 02:55:32,681 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=237 (was 285), ProcessCount=168 (was 171), AvailableMemoryMB=10366 (was 11156)