2023-04-29 22:53:32,077 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-29 22:53:32,428 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-29 22:53:32,546 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-29 22:53:32,608 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-29 22:53:32,617 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-29 22:53:32,632 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=170, AvailableMemoryMB=11378
2023-04-29 22:53:32,634 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-29 22:53:32,634 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-29 22:53:32,777 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1022222042:84 , srv1036980053:84 , srv1053813292:84 , srv1075906382:84 , srv1094046346:84 , srv1102688575:84 , srv1121076362:84 , srv1127534707:84 , srv1140311195:84 , srv1151350051:84 , srv115589145:84 , srv1186596527:84 , srv119097590:84 , srv1195866096:84 , srv121707699:84 , srv1224488234:84 , srv1249727671:84 , srv1266437603:84 , srv1299780530:84 , srv1362551946:84 , srv140503927:84 , srv1416202123:84 , srv1450534343:84 , srv1458083811:84 , srv1512934584:84 , srv1535401353:84 , srv1580620256:84 , srv1589321283:84 , srv1592113100:84 , srv1627399042:84 , srv1638295025:84 , srv1642468820:84 , srv1654391543:84 , srv1665645395:84 , srv1679013796:84 , srv1723173160:84 , srv1746403892:84 , srv1754911208:84 , srv1806816389:84 , srv1835230288:84 , srv1853732779:84 , srv1860719894:84 , srv1865347446:84 , srv1873826527:84 , srv1880435246:84 , srv1930148107:84 , srv1939294477:84 , srv198540705:84 , srv2008237243:84 , srv2018743099:84 , srv2044516803:84 , srv2044848077:84 , srv2070076015:84 , srv2077868290:84 , srv2098620618:84 , srv2104800929:84 , srv2138459283:84 , srv23214794:84 , srv256333881:84 , srv263723540:84 , srv274726695:84 , srv286708320:84 , srv351741451:84 , srv35414821:84 , srv391579368:84 , srv437022675:84 , srv44859623:84 , srv460829927:84 , srv501207596:84 , srv502242089:84 , srv540511601:84 , srv590662217:84 , srv613720005:84 , srv621896210:84 , srv627528471:84 , srv630580543:84 , srv679803152:84 , srv687736633:84 , srv704426655:84 , srv706434669:84 , srv745985810:84 , srv755983812:84 , srv767871669:84 , srv78981466:84 , srv790441242:84 , srv80924681:84 , srv809913400:84 , srv813123582:84 , srv822410966:84 , srv853246819:84 , srv871180424:84 , srv875769195:84 , srv915546451:84 , srv925437413:84 , srv934969394:84 , srv959594671:84 , srv967551515:84 , srv982533272:84 , srv987052363:84 , srv1948671386:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-29 22:53:32,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:53:32,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1224488234=15, srv44859623=67, srv80924681=86, srv115589145=10, srv822410966=89, srv959594671=96, srv1127534707=7, srv1930148107=45, srv198540705=48, srv501207596=69, srv2018743099=50, srv286708320=62, srv1723173160=35, srv1754911208=37, srv679803152=77, srv1140311195=8, srv967551515=97, srv1195866096=13, srv590662217=72, srv1939294477=46, srv1580620256=26, srv1102688575=5, srv263723540=60, srv627528471=75, srv2138459283=57, srv1094046346=4, srv1638295025=30, srv1654391543=32, srv755983812=82, srv1865347446=42, srv1880435246=44, srv2070076015=53, srv1806816389=38, srv1860719894=41, srv119097590=12, srv2044848077=52, srv1535401353=25, srv1458083811=23, srv875769195=92, srv1873826527=43, srv1022222042=0, srv140503927=20, srv1642468820=31, srv987052363=99, srv502242089=70, srv256333881=59, srv2008237243=49, srv1121076362=6, srv1627399042=29, srv351741451=63, srv121707699=14, srv745985810=81, srv1075906382=3, srv35414821=64, srv1036980053=1, srv2104800929=56, srv1853732779=40, srv2044516803=51, srv767871669=83, srv809913400=87, srv1362551946=19, srv813123582=88, srv1835230288=39, srv934969394=95, srv1589321283=27, srv1592113100=28, srv630580543=76, srv1416202123=21, srv1151350051=9, srv687736633=78, srv1186596527=11, srv2098620618=55, srv613720005=73, srv1266437603=17, srv621896210=74, srv1053813292=2, srv1299780530=18, srv704426655=79, srv1512934584=24, srv460829927=68, srv1948671386=47, srv540511601=71, srv853246819=90, srv915546451=93, srv274726695=61, srv391579368=65, srv871180424=91, srv437022675=66, srv23214794=58, srv790441242=85, srv925437413=94, srv1450534343=22, srv2077868290=54, srv1249727671=16, srv1746403892=36, srv78981466=84, srv706434669=80, srv1665645395=33, srv1679013796=34, srv982533272=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:53:32,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:53:32,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:53:32,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 22:53:32,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 22:53:32,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 22:53:32,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 22:53:32,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 22:53:32,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 22:53:32,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 22:53:32,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 22:53:32,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 22:53:32,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 22:53:33,137 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:33,137 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 22:53:33,138 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 22:53:33,526 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 599 ms to try 5000 different iterations.  Found a solution that moves 4692 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1796922255250984. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04061197895348144, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05692651848914681, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5213333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04061197895348144, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:33,537 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1036980053:82 , srv1627399042:82 , srv502242089:82 , srv706434669:82 , srv767871669:82 , srv1121076362:83 , srv1151350051:83 , srv115589145:83 , srv1638295025:83 , srv1939294477:83 , srv23214794:83 , srv967551515:83 , srv1186596527:84 , srv119097590:84 , srv1249727671:84 , srv1679013796:84 , srv2104800929:84 , srv274726695:84 , srv460829927:84 , srv621896210:84 , srv813123582:84 , srv982533272:84 , srv1102688575:85 , srv121707699:85 , srv1416202123:85 , srv1535401353:85 , srv1665645395:85 , srv1746403892:85 , srv1930148107:85 , srv2070076015:85 , srv35414821:85 , srv437022675:85 , srv44859623:85 , srv590662217:85 , srv704426655:85 , srv822410966:85 , srv853246819:85 , srv934969394:85 , srv1075906382:86 , srv1299780530:86 , srv1450534343:86 , srv1458083811:86 , srv1512934584:86 , srv1754911208:86 , srv1853732779:86 , srv198540705:86 , srv2077868290:86 , srv2138459283:86 , srv391579368:86 , srv687736633:86 , srv755983812:86 , srv78981466:86 , srv80924681:86 , srv987052363:86 , srv1022222042:87 , srv1195866096:87 , srv1266437603:87 , srv1362551946:87 , srv1592113100:87 , srv1654391543:87 , srv1723173160:87 , srv1865347446:87 , srv2044516803:87 , srv2044848077:87 , srv286708320:87 , srv501207596:87 , srv790441242:87 , srv871180424:87 , srv915546451:87 , srv1094046346:88 , srv1127534707:88 , srv1835230288:88 , srv1860719894:88 , srv1873826527:88 , srv1880435246:88 , srv2008237243:88 , srv2018743099:88 , srv2098620618:88 , srv351741451:88 , srv540511601:88 , srv613720005:88 , srv679803152:88 , srv745985810:88 , srv875769195:88 , srv959594671:88 , srv1053813292:89 , srv1806816389:89 , srv263723540:89 , srv809913400:89 , srv925437413:89 , srv1224488234:90 , srv1580620256:90 , srv1589321283:90 , srv630580543:90 , srv1642468820:91 , srv627528471:91 , srv1140311195:93 , srv140503927:93 , srv256333881:96 , srv1948671386:451 }
2023-04-29 22:53:33,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:53:33,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1224488234=15, srv44859623=67, srv80924681=86, srv115589145=10, srv822410966=89, srv959594671=96, srv1127534707=7, srv1930148107=45, srv198540705=48, srv501207596=69, srv2018743099=50, srv286708320=62, srv1723173160=35, srv1754911208=37, srv679803152=77, srv1140311195=8, srv967551515=97, srv1195866096=13, srv590662217=72, srv1939294477=46, srv1580620256=26, srv1102688575=5, srv263723540=60, srv627528471=75, srv2138459283=57, srv1094046346=4, srv1638295025=30, srv1654391543=32, srv755983812=82, srv1865347446=42, srv1880435246=44, srv2070076015=53, srv1806816389=38, srv1860719894=41, srv119097590=12, srv2044848077=52, srv1535401353=25, srv1458083811=23, srv875769195=92, srv1873826527=43, srv1022222042=0, srv140503927=20, srv1642468820=31, srv987052363=99, srv502242089=70, srv256333881=59, srv2008237243=49, srv1121076362=6, srv1627399042=29, srv351741451=63, srv121707699=14, srv745985810=81, srv1075906382=3, srv35414821=64, srv1036980053=1, srv2104800929=56, srv1853732779=40, srv2044516803=51, srv767871669=83, srv809913400=87, srv1362551946=19, srv813123582=88, srv1835230288=39, srv934969394=95, srv1589321283=27, srv1592113100=28, srv630580543=76, srv1416202123=21, srv1151350051=9, srv687736633=78, srv1186596527=11, srv2098620618=55, srv613720005=73, srv1266437603=17, srv621896210=74, srv1053813292=2, srv1299780530=18, srv704426655=79, srv1512934584=24, srv460829927=68, srv1948671386=47, srv540511601=71, srv853246819=90, srv915546451=93, srv274726695=61, srv391579368=65, srv871180424=91, srv437022675=66, srv23214794=58, srv790441242=85, srv925437413=94, srv1450534343=22, srv2077868290=54, srv1249727671=16, srv1746403892=36, srv78981466=84, srv706434669=80, srv1665645395=33, srv1679013796=34, srv982533272=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:53:33,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:53:33,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:53:33,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:53:33,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 22:53:33,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 22:53:33,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 22:53:33,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 22:53:33,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 22:53:33,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 22:53:33,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 22:53:33,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 22:53:33,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 22:53:33,696 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04061197895348144, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05692651848914681, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04061197895348144, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:33,696 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 22:53:33,696 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17965937143889393, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04061197895348144, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05692651848914681, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04061197895348144, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 22:53:34,016 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 427 ms to try 5000 different iterations.  Found a solution that moves 2679 regions; Going from a computed imbalance of 0.17965937143889393 to a new imbalance of 0.014342907807886992. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021410541170789867); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03259106210058205, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2976666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021410541170789867); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:34,030 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv256333881:74 , srv140503927:77 , srv1140311195:78 , srv1580620256:81 , srv1642468820:81 , srv630580543:81 , srv1806816389:82 , srv1835230288:82 , srv2018743099:82 , srv745985810:82 , srv1053813292:83 , srv1094046346:83 , srv1195866096:83 , srv1589321283:83 , srv2044516803:83 , srv286708320:83 , srv501207596:83 , srv871180424:83 , srv875769195:83 , srv925437413:83 , srv1224488234:84 , srv1860719894:84 , srv2077868290:84 , srv351741451:84 , srv391579368:84 , srv627528471:84 , srv679803152:84 , srv687736633:84 , srv1022222042:85 , srv1075906382:85 , srv1127534707:85 , srv1266437603:85 , srv1416202123:85 , srv1458083811:85 , srv1512934584:85 , srv1665645395:85 , srv1853732779:85 , srv1873826527:85 , srv2044848077:85 , srv2098620618:85 , srv2138459283:85 , srv35414821:85 , srv44859623:85 , srv80924681:85 , srv809913400:85 , srv915546451:85 , srv934969394:85 , srv1102688575:86 , srv1362551946:86 , srv1592113100:86 , srv1654391543:86 , srv1865347446:86 , srv198540705:86 , srv2070076015:86 , srv274726695:86 , srv540511601:86 , srv613720005:86 , srv755983812:86 , srv822410966:86 , srv115589145:87 , srv1186596527:87 , srv119097590:87 , srv1299780530:87 , srv1746403892:87 , srv1880435246:87 , srv2008237243:87 , srv437022675:87 , srv460829927:87 , srv621896210:87 , srv790441242:87 , srv813123582:87 , srv853246819:87 , srv959594671:87 , srv121707699:88 , srv1249727671:88 , srv1450534343:88 , srv1535401353:88 , srv1638295025:88 , srv1679013796:88 , srv1930148107:88 , srv590662217:88 , srv78981466:88 , srv982533272:88 , srv1151350051:89 , srv1627399042:89 , srv1723173160:89 , srv1754911208:89 , srv2104800929:89 , srv263723540:89 , srv704426655:89 , srv767871669:89 , srv1121076362:90 , srv1939294477:90 , srv23214794:90 , srv1036980053:91 , srv706434669:91 , srv967551515:91 , srv987052363:91 , srv502242089:94 , srv1948671386:513 }
2023-04-29 22:53:34,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:53:34,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1224488234=15, srv44859623=67, srv80924681=86, srv115589145=10, srv822410966=89, srv959594671=96, srv1127534707=7, srv1930148107=45, srv198540705=48, srv501207596=69, srv2018743099=50, srv286708320=62, srv1723173160=35, srv1754911208=37, srv679803152=77, srv1140311195=8, srv967551515=97, srv1195866096=13, srv590662217=72, srv1939294477=46, srv1580620256=26, srv1102688575=5, srv263723540=60, srv627528471=75, srv2138459283=57, srv1094046346=4, srv1638295025=30, srv1654391543=32, srv755983812=82, srv1865347446=42, srv1880435246=44, srv2070076015=53, srv1806816389=38, srv1860719894=41, srv119097590=12, srv2044848077=52, srv1535401353=25, srv1458083811=23, srv875769195=92, srv1873826527=43, srv1022222042=0, srv140503927=20, srv1642468820=31, srv987052363=99, srv502242089=70, srv256333881=59, srv2008237243=49, srv1121076362=6, srv1627399042=29, srv351741451=63, srv121707699=14, srv745985810=81, srv1075906382=3, srv35414821=64, srv1036980053=1, srv2104800929=56, srv1853732779=40, srv2044516803=51, srv767871669=83, srv809913400=87, srv1362551946=19, srv813123582=88, srv1835230288=39, srv934969394=95, srv1589321283=27, srv1592113100=28, srv630580543=76, srv1416202123=21, srv1151350051=9, srv687736633=78, srv1186596527=11, srv2098620618=55, srv613720005=73, srv1266437603=17, srv621896210=74, srv1053813292=2, srv1299780530=18, srv704426655=79, srv1512934584=24, srv460829927=68, srv1948671386=47, srv540511601=71, srv853246819=90, srv915546451=93, srv274726695=61, srv391579368=65, srv871180424=91, srv437022675=66, srv23214794=58, srv790441242=85, srv925437413=94, srv1450534343=22, srv2077868290=54, srv1249727671=16, srv1746403892=36, srv78981466=84, srv706434669=80, srv1665645395=33, srv1679013796=34, srv982533272=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:53:34,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:53:34,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:53:34,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:53:34,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:53:34,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 22:53:34,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 22:53:34,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 22:53:34,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 22:53:34,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 22:53:34,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 22:53:34,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 22:53:34,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 22:53:34,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 22:53:34,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 22:53:34,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 22:53:34,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 22:53:34,215 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021410541170789867); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03259106210058205, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021410541170789867); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:34,215 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 22:53:34,215 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014324149048947998, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021410541170789867); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03259106210058205, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021410541170789867); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 22:53:34,523 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations.  Found a solution that moves 952 regions; Going from a computed imbalance of 0.014324149048947998 to a new imbalance of 5.296689408823853E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005418051421602886); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008723203316114791); MoveCostFunction : (multiplier=7.0, imbalance=0.10577777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005418051421602886); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0025833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:34,525 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv263723540:80 , srv1723173160:81 , srv502242089:81 , srv987052363:81 , srv1754911208:82 , srv1450534343:83 , srv1880435246:83 , srv2008237243:83 , srv613720005:83 , srv627528471:83 , srv959594671:83 , srv967551515:83 , srv1121076362:84 , srv1127534707:84 , srv121707699:84 , srv1224488234:84 , srv1873826527:84 , srv1930148107:84 , srv2104800929:84 , srv23214794:84 , srv540511601:84 , srv704426655:84 , srv78981466:84 , srv790441242:84 , srv809913400:84 , srv1022222042:85 , srv1036980053:85 , srv1151350051:85 , srv1266437603:85 , srv1299780530:85 , srv1362551946:85 , srv1535401353:85 , srv1589321283:85 , srv1592113100:85 , srv1654391543:85 , srv1865347446:85 , srv1939294477:85 , srv2098620618:85 , srv2138459283:85 , srv351741451:85 , srv437022675:85 , srv590662217:85 , srv679803152:85 , srv706434669:85 , srv853246819:85 , srv915546451:85 , srv982533272:85 , srv1053813292:86 , srv1075906382:86 , srv1094046346:86 , srv1102688575:86 , srv1140311195:86 , srv1249727671:86 , srv1512934584:86 , srv1627399042:86 , srv1642468820:86 , srv1679013796:86 , srv1746403892:86 , srv1806816389:86 , srv1860719894:86 , srv198540705:86 , srv2044848077:86 , srv2070076015:86 , srv621896210:86 , srv630580543:86 , srv755983812:86 , srv767871669:86 , srv925437413:86 , srv115589145:87 , srv1186596527:87 , srv119097590:87 , srv1195866096:87 , srv140503927:87 , srv1458083811:87 , srv1580620256:87 , srv1638295025:87 , srv1853732779:87 , srv2018743099:87 , srv2044516803:87 , srv256333881:87 , srv274726695:87 , srv286708320:87 , srv35414821:87 , srv44859623:87 , srv460829927:87 , srv501207596:87 , srv687736633:87 , srv80924681:87 , srv813123582:87 , srv822410966:87 , srv871180424:87 , srv875769195:87 , srv1416202123:88 , srv1665645395:88 , srv1835230288:88 , srv2077868290:88 , srv391579368:88 , srv745985810:88 , srv934969394:88 , srv1948671386:542 }
2023-04-29 22:53:34,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:53:34,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1224488234=15, srv44859623=67, srv80924681=86, srv115589145=10, srv822410966=89, srv959594671=96, srv1127534707=7, srv1930148107=45, srv198540705=48, srv501207596=69, srv2018743099=50, srv286708320=62, srv1723173160=35, srv1754911208=37, srv679803152=77, srv1140311195=8, srv967551515=97, srv1195866096=13, srv590662217=72, srv1939294477=46, srv1580620256=26, srv1102688575=5, srv263723540=60, srv627528471=75, srv2138459283=57, srv1094046346=4, srv1638295025=30, srv1654391543=32, srv755983812=82, srv1865347446=42, srv1880435246=44, srv2070076015=53, srv1806816389=38, srv1860719894=41, srv119097590=12, srv2044848077=52, srv1535401353=25, srv1458083811=23, srv875769195=92, srv1873826527=43, srv1022222042=0, srv140503927=20, srv1642468820=31, srv987052363=99, srv502242089=70, srv256333881=59, srv2008237243=49, srv1121076362=6, srv1627399042=29, srv351741451=63, srv121707699=14, srv745985810=81, srv1075906382=3, srv35414821=64, srv1036980053=1, srv2104800929=56, srv1853732779=40, srv2044516803=51, srv767871669=83, srv809913400=87, srv1362551946=19, srv813123582=88, srv1835230288=39, srv934969394=95, srv1589321283=27, srv1592113100=28, srv630580543=76, srv1416202123=21, srv1151350051=9, srv687736633=78, srv1186596527=11, srv2098620618=55, srv613720005=73, srv1266437603=17, srv621896210=74, srv1053813292=2, srv1299780530=18, srv704426655=79, srv1512934584=24, srv460829927=68, srv1948671386=47, srv540511601=71, srv853246819=90, srv915546451=93, srv274726695=61, srv391579368=65, srv871180424=91, srv437022675=66, srv23214794=58, srv790441242=85, srv925437413=94, srv1450534343=22, srv2077868290=54, srv1249727671=16, srv1746403892=36, srv78981466=84, srv706434669=80, srv1665645395=33, srv1679013796=34, srv982533272=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:53:34,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:53:34,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 22:53:34,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 22:53:34,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 22:53:34,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 22:53:34,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 22:53:34,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 22:53:34,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 22:53:34,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 22:53:34,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 22:53:34,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 22:53:34,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 22:53:34,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 22:53:34,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 22:53:34,663 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005418051421602886); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008723203316114791); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005418051421602886); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0025833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:34,663 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 22:53:34,664 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.23002894406117E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005418051421602886); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008723203316114791); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005418051421602886); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0025833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 22:53:35,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 593 ms to try 5000 different iterations.  Found a solution that moves 233 regions; Going from a computed imbalance of 5.23002894406117E-4 to a new imbalance of 2.0442350908091333E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.02588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:35,158 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1665645395:83 , srv1835230288:83 , srv934969394:83 , srv1036980053:84 , srv1053813292:84 , srv1186596527:84 , srv119097590:84 , srv1224488234:84 , srv1249727671:84 , srv1299780530:84 , srv1362551946:84 , srv140503927:84 , srv1416202123:84 , srv1458083811:84 , srv1535401353:84 , srv1580620256:84 , srv1589321283:84 , srv1592113100:84 , srv1638295025:84 , srv1642468820:84 , srv1679013796:84 , srv1746403892:84 , srv1853732779:84 , srv1860719894:84 , srv1865347446:84 , srv1880435246:84 , srv1939294477:84 , srv198540705:84 , srv2008237243:84 , srv2044848077:84 , srv2077868290:84 , srv2098620618:84 , srv263723540:84 , srv391579368:84 , srv460829927:84 , srv540511601:84 , srv590662217:84 , srv687736633:84 , srv704426655:84 , srv706434669:84 , srv767871669:84 , srv78981466:84 , srv790441242:84 , srv80924681:84 , srv809913400:84 , srv822410966:84 , srv871180424:84 , srv925437413:84 , srv959594671:84 , srv987052363:84 , srv1022222042:85 , srv1075906382:85 , srv1094046346:85 , srv1121076362:85 , srv1127534707:85 , srv1140311195:85 , srv1151350051:85 , srv115589145:85 , srv1195866096:85 , srv121707699:85 , srv1266437603:85 , srv1450534343:85 , srv1512934584:85 , srv1627399042:85 , srv1654391543:85 , srv1723173160:85 , srv1754911208:85 , srv1806816389:85 , srv1873826527:85 , srv1930148107:85 , srv2018743099:85 , srv2044516803:85 , srv2070076015:85 , srv2104800929:85 , srv23214794:85 , srv256333881:85 , srv274726695:85 , srv286708320:85 , srv351741451:85 , srv35414821:85 , srv437022675:85 , srv44859623:85 , srv501207596:85 , srv502242089:85 , srv613720005:85 , srv621896210:85 , srv627528471:85 , srv630580543:85 , srv679803152:85 , srv745985810:85 , srv755983812:85 , srv813123582:85 , srv853246819:85 , srv875769195:85 , srv915546451:85 , srv967551515:85 , srv982533272:85 , srv1102688575:86 , srv2138459283:86 , srv1948671386:636 }
2023-04-29 22:53:35,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:53:35,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1224488234=15, srv44859623=67, srv80924681=86, srv115589145=10, srv822410966=89, srv959594671=96, srv1127534707=7, srv1930148107=45, srv198540705=48, srv501207596=69, srv2018743099=50, srv286708320=62, srv1723173160=35, srv1754911208=37, srv679803152=77, srv1140311195=8, srv967551515=97, srv1195866096=13, srv590662217=72, srv1939294477=46, srv1580620256=26, srv1102688575=5, srv263723540=60, srv627528471=75, srv2138459283=57, srv1094046346=4, srv1638295025=30, srv1654391543=32, srv755983812=82, srv1865347446=42, srv1880435246=44, srv2070076015=53, srv1806816389=38, srv1860719894=41, srv119097590=12, srv2044848077=52, srv1535401353=25, srv1458083811=23, srv875769195=92, srv1873826527=43, srv1022222042=0, srv140503927=20, srv1642468820=31, srv987052363=99, srv502242089=70, srv256333881=59, srv2008237243=49, srv1121076362=6, srv1627399042=29, srv351741451=63, srv121707699=14, srv745985810=81, srv1075906382=3, srv35414821=64, srv1036980053=1, srv2104800929=56, srv1853732779=40, srv2044516803=51, srv767871669=83, srv809913400=87, srv1362551946=19, srv813123582=88, srv1835230288=39, srv934969394=95, srv1589321283=27, srv1592113100=28, srv630580543=76, srv1416202123=21, srv1151350051=9, srv687736633=78, srv1186596527=11, srv2098620618=55, srv613720005=73, srv1266437603=17, srv621896210=74, srv1053813292=2, srv1299780530=18, srv704426655=79, srv1512934584=24, srv460829927=68, srv1948671386=47, srv540511601=71, srv853246819=90, srv915546451=93, srv274726695=61, srv391579368=65, srv871180424=91, srv437022675=66, srv23214794=58, srv790441242=85, srv925437413=94, srv1450534343=22, srv2077868290=54, srv1249727671=16, srv1746403892=36, srv78981466=84, srv706434669=80, srv1665645395=33, srv1679013796=34, srv982533272=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:53:35,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:53:35,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:53:35,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:53:35,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:53:35,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:53:35,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:53:35,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:53:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 22:53:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 22:53:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 22:53:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 22:53:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 22:53:35,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.881085003732401E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:35,289 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1665645395:83 , srv1835230288:83 , srv934969394:83 , srv1036980053:84 , srv1053813292:84 , srv1186596527:84 , srv119097590:84 , srv1224488234:84 , srv1249727671:84 , srv1299780530:84 , srv1362551946:84 , srv140503927:84 , srv1416202123:84 , srv1458083811:84 , srv1535401353:84 , srv1580620256:84 , srv1589321283:84 , srv1592113100:84 , srv1638295025:84 , srv1642468820:84 , srv1679013796:84 , srv1746403892:84 , srv1853732779:84 , srv1860719894:84 , srv1865347446:84 , srv1880435246:84 , srv1939294477:84 , srv198540705:84 , srv2008237243:84 , srv2044848077:84 , srv2077868290:84 , srv2098620618:84 , srv263723540:84 , srv391579368:84 , srv460829927:84 , srv540511601:84 , srv590662217:84 , srv687736633:84 , srv704426655:84 , srv706434669:84 , srv767871669:84 , srv78981466:84 , srv790441242:84 , srv80924681:84 , srv809913400:84 , srv822410966:84 , srv871180424:84 , srv925437413:84 , srv959594671:84 , srv987052363:84 , srv1022222042:85 , srv1075906382:85 , srv1094046346:85 , srv1121076362:85 , srv1127534707:85 , srv1140311195:85 , srv1151350051:85 , srv115589145:85 , srv1195866096:85 , srv121707699:85 , srv1266437603:85 , srv1450534343:85 , srv1512934584:85 , srv1627399042:85 , srv1654391543:85 , srv1723173160:85 , srv1754911208:85 , srv1806816389:85 , srv1873826527:85 , srv1930148107:85 , srv2018743099:85 , srv2044516803:85 , srv2070076015:85 , srv2104800929:85 , srv23214794:85 , srv256333881:85 , srv274726695:85 , srv286708320:85 , srv351741451:85 , srv35414821:85 , srv437022675:85 , srv44859623:85 , srv501207596:85 , srv502242089:85 , srv613720005:85 , srv621896210:85 , srv627528471:85 , srv630580543:85 , srv679803152:85 , srv745985810:85 , srv755983812:85 , srv813123582:85 , srv853246819:85 , srv875769195:85 , srv915546451:85 , srv967551515:85 , srv982533272:85 , srv1102688575:86 , srv2138459283:86 , srv1948671386:636 }
2023-04-29 22:53:35,317 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=359 (was 338) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10578 (was 11378)
2023-04-29 22:53:35,327 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=359, ProcessCount=170, AvailableMemoryMB=10578
2023-04-29 22:53:35,328 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-29 22:53:35,328 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-29 22:53:35,343 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1072667674:84 , srv1088380434:84 , srv1159863487:84 , srv1164286674:84 , srv1265426125:84 , srv1281541818:84 , srv1326706401:84 , srv1348290035:84 , srv1361294410:84 , srv1368746683:84 , srv1384896062:84 , srv1481080377:84 , srv159940122:84 , srv1631104305:84 , srv171774233:84 , srv1825008420:84 , srv1868624686:84 , srv1950543343:84 , srv2109115000:84 , srv2115640091:84 , srv2120307088:84 , srv307417:84 , srv529066047:84 , srv584552237:84 , srv586970039:84 , srv686360650:84 , srv702522730:84 , srv717450616:84 , srv796146449:84 , srv463648324:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-29 22:53:35,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:53:35,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481080377=11, srv2115640091=19, srv1326706401=6, srv463648324=22, srv529066047=23, srv702522730=27, srv1361294410=8, srv1368746683=9, srv2109115000=18, srv1159863487=2, srv2120307088=20, srv1868624686=16, srv796146449=29, srv1265426125=4, srv1950543343=17, srv1384896062=10, srv159940122=12, srv1825008420=15, srv717450616=28, srv1164286674=3, srv171774233=14, srv686360650=26, srv1088380434=1, srv1281541818=5, srv1631104305=13, srv1072667674=0, srv584552237=24, srv1348290035=7, srv586970039=25, srv307417=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:53:35,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:53:35,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:53:35,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:53:35,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:53:35,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:53:35,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:53:35,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:53:35,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-29 22:53:35,385 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0729038024271322, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:53:35,385 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.99124227607652, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0729038024271322, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-29 22:55:35,366 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6128628 different iterations.  Found a solution that moves 1856 regions; Going from a computed imbalance of 0.99124227607652 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:55:35,373 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1072667674:90 , srv1088380434:90 , srv1159863487:90 , srv1164286674:90 , srv1265426125:90 , srv1281541818:90 , srv1326706401:90 , srv1348290035:90 , srv1361294410:90 , srv1368746683:90 , srv1384896062:90 , srv1481080377:90 , srv159940122:90 , srv1631104305:90 , srv171774233:90 , srv1825008420:90 , srv1868624686:90 , srv1950543343:90 , srv2109115000:90 , srv2115640091:90 , srv2120307088:90 , srv307417:90 , srv463648324:90 , srv529066047:90 , srv584552237:90 , srv586970039:90 , srv686360650:90 , srv702522730:90 , srv717450616:90 , srv796146449:90 }
2023-04-29 22:55:35,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 22:55:35,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481080377=11, srv2115640091=19, srv1326706401=6, srv463648324=22, srv529066047=23, srv702522730=27, srv1361294410=8, srv1368746683=9, srv2109115000=18, srv1159863487=2, srv2120307088=20, srv1868624686=16, srv796146449=29, srv1265426125=4, srv1950543343=17, srv1384896062=10, srv159940122=12, srv1825008420=15, srv717450616=28, srv1164286674=3, srv171774233=14, srv686360650=26, srv1088380434=1, srv1281541818=5, srv1631104305=13, srv1072667674=0, srv584552237=24, srv1348290035=7, srv586970039=25, srv307417=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 22:55:35,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 22:55:35,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-29 22:55:35,409 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 22:55:35,410 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1072667674:90 , srv1088380434:90 , srv1159863487:90 , srv1164286674:90 , srv1265426125:90 , srv1281541818:90 , srv1326706401:90 , srv1348290035:90 , srv1361294410:90 , srv1368746683:90 , srv1384896062:90 , srv1481080377:90 , srv159940122:90 , srv1631104305:90 , srv171774233:90 , srv1825008420:90 , srv1868624686:90 , srv1950543343:90 , srv2109115000:90 , srv2115640091:90 , srv2120307088:90 , srv307417:90 , srv463648324:90 , srv529066047:90 , srv584552237:90 , srv586970039:90 , srv686360650:90 , srv702522730:90 , srv717450616:90 , srv796146449:90 }
2023-04-29 22:55:35,428 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=172 (was 359), ProcessCount=170 (was 170), AvailableMemoryMB=10705 (was 10578) - AvailableMemoryMB LEAK? -