2023-11-02 03:55:28,286 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-02 03:55:28,580 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-02 03:55:28,682 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-02 03:55:28,740 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-02 03:55:28,750 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-02 03:55:28,765 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=89, ProcessCount=167, AvailableMemoryMB=11994
2023-11-02 03:55:28,767 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-02 03:55:28,767 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-02 03:55:28,891 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007925750:84 , srv1026680360:84 , srv1037524735:84 , srv1066783921:84 , srv1091030980:84 , srv1108423188:84 , srv1183126045:84 , srv1189600337:84 , srv1190747604:84 , srv1198060599:84 , srv1216714115:84 , srv122741260:84 , srv1261948230:84 , srv127599620:84 , srv1276636385:84 , srv12951872:84 , srv1322633224:84 , srv1341256932:84 , srv13521891:84 , srv1376251021:84 , srv1376515659:84 , srv1383682517:84 , srv1401871506:84 , srv140751571:84 , srv1521177070:84 , srv1528136329:84 , srv1530404489:84 , srv1538140446:84 , srv1543129448:84 , srv1547904543:84 , srv1555207078:84 , srv1576542483:84 , srv1579371166:84 , srv1597602939:84 , srv1655672908:84 , srv1664298695:84 , srv1674908428:84 , srv1690712328:84 , srv1701897866:84 , srv1718096565:84 , srv1719384290:84 , srv1748192447:84 , srv177931904:84 , srv1786641738:84 , srv1802523675:84 , srv1808838765:84 , srv1853234732:84 , srv1864578648:84 , srv1892914015:84 , srv1899721301:84 , srv1985446782:84 , srv2003559001:84 , srv2019399608:84 , srv2027164122:84 , srv2029785947:84 , srv2049461009:84 , srv2050613308:84 , srv2063369666:84 , srv2071020520:84 , srv2093812916:84 , srv216964404:84 , srv224566555:84 , srv240139598:84 , srv256806816:84 , srv270835944:84 , srv290141139:84 , srv331359738:84 , srv352242868:84 , srv378399700:84 , srv380533046:84 , srv385314004:84 , srv48973133:84 , srv496206470:84 , srv539307596:84 , srv545403496:84 , srv566134612:84 , srv56978064:84 , srv570463511:84 , srv579837719:84 , srv58568221:84 , srv595301845:84 , srv599348959:84 , srv634389735:84 , srv64163809:84 , srv661057723:84 , srv664769527:84 , srv68751380:84 , srv688190105:84 , srv714011844:84 , srv723854033:84 , srv744476660:84 , srv778962024:84 , srv798882542:84 , srv846476897:84 , srv850742511:84 , srv85701483:84 , srv86290655:84 , srv87442571:84 , srv915752569:84 , srv780957070:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-02 03:55:28,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:28,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1376251021=19, srv256806816=63, srv68751380=86, srv1190747604=8, srv1108423188=5, srv122741260=11, srv1401871506=22, srv1341256932=17, srv270835944=64, srv13521891=18, srv545403496=74, srv2071020520=58, srv1189600337=7, srv1037524735=2, srv566134612=75, srv1198060599=9, srv1547904543=29, srv1576542483=31, srv846476897=94, srv2003559001=51, srv661057723=84, srv140751571=23, srv915752569=99, srv86290655=97, srv2050613308=56, srv56978064=76, srv664769527=85, srv352242868=67, srv1853234732=46, srv331359738=66, srv1555207078=30, srv1664298695=35, srv1718096565=39, srv1538140446=27, srv1183126045=6, srv780957070=92, srv58568221=79, srv1674908428=36, srv1748192447=41, srv634389735=82, srv1007925750=0, srv378399700=68, srv1276636385=14, srv723854033=89, srv1579371166=32, srv688190105=87, srv798882542=93, srv850742511=95, srv1026680360=1, srv216964404=60, srv1864578648=47, srv1528136329=25, srv2049461009=55, srv224566555=61, srv1690712328=37, srv599348959=81, srv744476660=90, srv85701483=96, srv1521177070=24, srv778962024=91, srv1802523675=44, srv1655672908=34, srv1322633224=16, srv290141139=65, srv1808838765=45, srv714011844=88, srv48973133=71, srv2027164122=53, srv1786641738=43, srv380533046=69, srv1543129448=28, srv1597602939=33, srv570463511=77, srv1701897866=38, srv240139598=62, srv1066783921=3, srv579837719=78, srv595301845=80, srv1719384290=40, srv2029785947=54, srv539307596=73, srv2019399608=52, srv1376515659=20, srv177931904=42, srv1985446782=50, srv2093812916=59, srv1261948230=12, srv1216714115=10, srv12951872=15, srv64163809=83, srv127599620=13, srv496206470=72, srv385314004=70, srv1530404489=26, srv2063369666=57, srv1091030980=4, srv87442571=98, srv1383682517=21, srv1892914015=48, srv1899721301=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:29,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:29,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 03:55:29,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 03:55:29,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 03:55:29,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 03:55:29,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 03:55:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 03:55:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 03:55:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 03:55:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 03:55:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 03:55:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 03:55:29,271 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:29,272 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 03:55:29,272 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 03:55:29,678 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 644 ms to try 5000 different iterations.  Found a solution that moves 4718 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1728558498756991. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039817516156108984, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05567461988791032, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5242222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039817516156108984, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16733333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:29,688 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv58568221:82 , srv1007925750:83 , srv1189600337:83 , srv1748192447:83 , srv380533046:83 , srv545403496:83 , srv664769527:83 , srv68751380:83 , srv1341256932:84 , srv1383682517:84 , srv1528136329:84 , srv1576542483:84 , srv177931904:84 , srv2019399608:84 , srv2027164122:84 , srv216964404:84 , srv579837719:84 , srv64163809:84 , srv850742511:84 , srv87442571:84 , srv1026680360:85 , srv1276636385:85 , srv1376251021:85 , srv1401871506:85 , srv1547904543:85 , srv1597602939:85 , srv1674908428:85 , srv1690712328:85 , srv1701897866:85 , srv1899721301:85 , srv2049461009:85 , srv2050613308:85 , srv2093812916:85 , srv352242868:85 , srv496206470:85 , srv56978064:85 , srv723854033:85 , srv798882542:85 , srv86290655:85 , srv1190747604:86 , srv122741260:86 , srv127599620:86 , srv1376515659:86 , srv140751571:86 , srv1538140446:86 , srv1555207078:86 , srv1579371166:86 , srv1655672908:86 , srv1802523675:86 , srv1853234732:86 , srv256806816:86 , srv566134612:86 , srv599348959:86 , srv714011844:86 , srv846476897:86 , srv85701483:86 , srv1066783921:87 , srv1183126045:87 , srv1216714115:87 , srv12951872:87 , srv1530404489:87 , srv1864578648:87 , srv1985446782:87 , srv2003559001:87 , srv2029785947:87 , srv331359738:87 , srv378399700:87 , srv48973133:87 , srv539307596:87 , srv570463511:87 , srv661057723:87 , srv744476660:87 , srv1091030980:88 , srv13521891:88 , srv1786641738:88 , srv240139598:88 , srv290141139:88 , srv385314004:88 , srv595301845:88 , srv778962024:88 , srv1037524735:89 , srv1322633224:89 , srv1543129448:89 , srv1719384290:89 , srv1808838765:89 , srv2063369666:89 , srv2071020520:89 , srv224566555:89 , srv634389735:89 , srv915752569:89 , srv1108423188:90 , srv1198060599:90 , srv1261948230:90 , srv1521177070:90 , srv688190105:90 , srv1664298695:92 , srv1892914015:92 , srv270835944:92 , srv1718096565:96 , srv780957070:444 }
2023-11-02 03:55:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1376251021=19, srv256806816=63, srv68751380=86, srv1190747604=8, srv1108423188=5, srv122741260=11, srv1401871506=22, srv1341256932=17, srv270835944=64, srv13521891=18, srv545403496=74, srv2071020520=58, srv1189600337=7, srv1037524735=2, srv566134612=75, srv1198060599=9, srv1547904543=29, srv1576542483=31, srv846476897=94, srv2003559001=51, srv661057723=84, srv140751571=23, srv915752569=99, srv86290655=97, srv2050613308=56, srv56978064=76, srv664769527=85, srv352242868=67, srv1853234732=46, srv331359738=66, srv1555207078=30, srv1664298695=35, srv1718096565=39, srv1538140446=27, srv1183126045=6, srv780957070=92, srv58568221=79, srv1674908428=36, srv1748192447=41, srv634389735=82, srv1007925750=0, srv378399700=68, srv1276636385=14, srv723854033=89, srv1579371166=32, srv688190105=87, srv798882542=93, srv850742511=95, srv1026680360=1, srv216964404=60, srv1864578648=47, srv1528136329=25, srv2049461009=55, srv224566555=61, srv1690712328=37, srv599348959=81, srv744476660=90, srv85701483=96, srv1521177070=24, srv778962024=91, srv1802523675=44, srv1655672908=34, srv1322633224=16, srv290141139=65, srv1808838765=45, srv714011844=88, srv48973133=71, srv2027164122=53, srv1786641738=43, srv380533046=69, srv1543129448=28, srv1597602939=33, srv570463511=77, srv1701897866=38, srv240139598=62, srv1066783921=3, srv579837719=78, srv595301845=80, srv1719384290=40, srv2029785947=54, srv539307596=73, srv2019399608=52, srv1376515659=20, srv177931904=42, srv1985446782=50, srv2093812916=59, srv1261948230=12, srv1216714115=10, srv12951872=15, srv64163809=83, srv127599620=13, srv496206470=72, srv385314004=70, srv1530404489=26, srv2063369666=57, srv1091030980=4, srv87442571=98, srv1383682517=21, srv1892914015=48, srv1899721301=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:29,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:29,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 03:55:29,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 03:55:29,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 03:55:29,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 03:55:29,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 03:55:29,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 03:55:29,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 03:55:29,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 03:55:29,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 03:55:29,874 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039817516156108984, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05567461988791032, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039817516156108984, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16733333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:29,874 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 03:55:29,875 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17282281373360348, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039817516156108984, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05567461988791032, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039817516156108984, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16733333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 03:55:30,156 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 422 ms to try 5000 different iterations.  Found a solution that moves 2565 regions; Going from a computed imbalance of 0.17282281373360348 to a new imbalance of 0.015636932477490657. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023961664523858922); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035844783230596956, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.285, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023961664523858922); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:30,165 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1718096565:75 , srv1892914015:78 , srv270835944:79 , srv1664298695:80 , srv1198060599:81 , srv1521177070:81 , srv1808838765:81 , srv2063369666:82 , srv224566555:82 , srv48973133:82 , srv1066783921:83 , srv12951872:83 , srv1322633224:83 , srv1538140446:83 , srv1719384290:83 , srv1786641738:83 , srv1864578648:83 , srv2071020520:83 , srv385314004:83 , srv915752569:83 , srv1108423188:84 , srv1216714115:84 , srv122741260:84 , srv13521891:84 , srv1543129448:84 , srv1802523675:84 , srv2003559001:84 , srv240139598:84 , srv256806816:84 , srv290141139:84 , srv378399700:84 , srv539307596:84 , srv634389735:84 , srv688190105:84 , srv778962024:84 , srv1026680360:85 , srv127599620:85 , srv1376515659:85 , srv140751571:85 , srv1597602939:85 , srv1853234732:85 , srv1899721301:85 , srv566134612:85 , srv595301845:85 , srv714011844:85 , srv744476660:85 , srv86290655:85 , srv1037524735:86 , srv1091030980:86 , srv1183126045:86 , srv1190747604:86 , srv1261948230:86 , srv1401871506:86 , srv1530404489:86 , srv1579371166:86 , srv1655672908:86 , srv1701897866:86 , srv1985446782:86 , srv2027164122:86 , srv2049461009:86 , srv2050613308:86 , srv2093812916:86 , srv216964404:86 , srv570463511:86 , srv661057723:86 , srv846476897:86 , srv850742511:86 , srv1376251021:87 , srv1555207078:87 , srv1576542483:87 , srv1690712328:87 , srv331359738:87 , srv380533046:87 , srv496206470:87 , srv56978064:87 , srv599348959:87 , srv664769527:87 , srv723854033:87 , srv85701483:87 , srv87442571:87 , srv1341256932:88 , srv1547904543:88 , srv177931904:88 , srv2029785947:88 , srv798882542:88 , srv1007925750:89 , srv1189600337:89 , srv1276636385:89 , srv1383682517:89 , srv1674908428:89 , srv579837719:89 , srv58568221:89 , srv64163809:89 , srv1528136329:90 , srv352242868:90 , srv545403496:90 , srv1748192447:91 , srv2019399608:91 , srv68751380:91 , srv780957070:543 }
2023-11-02 03:55:30,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:30,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1376251021=19, srv256806816=63, srv68751380=86, srv1190747604=8, srv1108423188=5, srv122741260=11, srv1401871506=22, srv1341256932=17, srv270835944=64, srv13521891=18, srv545403496=74, srv2071020520=58, srv1189600337=7, srv1037524735=2, srv566134612=75, srv1198060599=9, srv1547904543=29, srv1576542483=31, srv846476897=94, srv2003559001=51, srv661057723=84, srv140751571=23, srv915752569=99, srv86290655=97, srv2050613308=56, srv56978064=76, srv664769527=85, srv352242868=67, srv1853234732=46, srv331359738=66, srv1555207078=30, srv1664298695=35, srv1718096565=39, srv1538140446=27, srv1183126045=6, srv780957070=92, srv58568221=79, srv1674908428=36, srv1748192447=41, srv634389735=82, srv1007925750=0, srv378399700=68, srv1276636385=14, srv723854033=89, srv1579371166=32, srv688190105=87, srv798882542=93, srv850742511=95, srv1026680360=1, srv216964404=60, srv1864578648=47, srv1528136329=25, srv2049461009=55, srv224566555=61, srv1690712328=37, srv599348959=81, srv744476660=90, srv85701483=96, srv1521177070=24, srv778962024=91, srv1802523675=44, srv1655672908=34, srv1322633224=16, srv290141139=65, srv1808838765=45, srv714011844=88, srv48973133=71, srv2027164122=53, srv1786641738=43, srv380533046=69, srv1543129448=28, srv1597602939=33, srv570463511=77, srv1701897866=38, srv240139598=62, srv1066783921=3, srv579837719=78, srv595301845=80, srv1719384290=40, srv2029785947=54, srv539307596=73, srv2019399608=52, srv1376515659=20, srv177931904=42, srv1985446782=50, srv2093812916=59, srv1261948230=12, srv1216714115=10, srv12951872=15, srv64163809=83, srv127599620=13, srv496206470=72, srv385314004=70, srv1530404489=26, srv2063369666=57, srv1091030980=4, srv87442571=98, srv1383682517=21, srv1892914015=48, srv1899721301=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 03:55:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 03:55:30,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 03:55:30,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 03:55:30,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 03:55:30,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 03:55:30,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 03:55:30,308 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023961664523858922); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035844783230596956, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023961664523858922); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:30,308 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 03:55:30,308 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015618971963612897, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023961664523858922); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035844783230596956, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023961664523858922); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 03:55:30,556 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 347 ms to try 5000 different iterations.  Found a solution that moves 1007 regions; Going from a computed imbalance of 0.015618971963612897 to a new imbalance of 6.71079769903752E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0088000612180301); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009767211922128139); MoveCostFunction : (multiplier=7.0, imbalance=0.11188888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0088000612180301); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:30,558 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037524735:82 , srv1091030980:82 , srv1261948230:82 , srv1748192447:82 , srv2019399608:82 , srv2029785947:82 , srv352242868:82 , srv1183126045:83 , srv1383682517:83 , srv1528136329:83 , srv1530404489:83 , srv1674908428:83 , srv634389735:83 , srv661057723:83 , srv1276636385:84 , srv13521891:84 , srv1543129448:84 , srv1547904543:84 , srv1555207078:84 , srv1664298695:84 , srv1985446782:84 , srv331359738:84 , srv545403496:84 , srv570463511:84 , srv595301845:84 , srv599348959:84 , srv68751380:84 , srv688190105:84 , srv846476897:84 , srv85701483:84 , srv1007925750:85 , srv1108423188:85 , srv1190747604:85 , srv1322633224:85 , srv1579371166:85 , srv1655672908:85 , srv1701897866:85 , srv2050613308:85 , srv2071020520:85 , srv240139598:85 , srv566134612:85 , srv579837719:85 , srv744476660:85 , srv798882542:85 , srv915752569:85 , srv1026680360:86 , srv1189600337:86 , srv1216714115:86 , srv122741260:86 , srv127599620:86 , srv12951872:86 , srv1401871506:86 , srv140751571:86 , srv1521177070:86 , srv1690712328:86 , srv1718096565:86 , srv1719384290:86 , srv1802523675:86 , srv1853234732:86 , srv1864578648:86 , srv1892914015:86 , srv2003559001:86 , srv224566555:86 , srv270835944:86 , srv290141139:86 , srv378399700:86 , srv385314004:86 , srv496206470:86 , srv539307596:86 , srv56978064:86 , srv58568221:86 , srv64163809:86 , srv714011844:86 , srv723854033:86 , srv778962024:86 , srv1066783921:87 , srv1198060599:87 , srv1341256932:87 , srv1376251021:87 , srv1376515659:87 , srv1576542483:87 , srv177931904:87 , srv1786641738:87 , srv1808838765:87 , srv2049461009:87 , srv2063369666:87 , srv2093812916:87 , srv216964404:87 , srv256806816:87 , srv380533046:87 , srv86290655:87 , srv87442571:87 , srv1597602939:88 , srv1899721301:88 , srv2027164122:88 , srv664769527:88 , srv850742511:88 , srv1538140446:89 , srv48973133:89 , srv780957070:549 }
2023-11-02 03:55:30,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:30,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1376251021=19, srv256806816=63, srv68751380=86, srv1190747604=8, srv1108423188=5, srv122741260=11, srv1401871506=22, srv1341256932=17, srv270835944=64, srv13521891=18, srv545403496=74, srv2071020520=58, srv1189600337=7, srv1037524735=2, srv566134612=75, srv1198060599=9, srv1547904543=29, srv1576542483=31, srv846476897=94, srv2003559001=51, srv661057723=84, srv140751571=23, srv915752569=99, srv86290655=97, srv2050613308=56, srv56978064=76, srv664769527=85, srv352242868=67, srv1853234732=46, srv331359738=66, srv1555207078=30, srv1664298695=35, srv1718096565=39, srv1538140446=27, srv1183126045=6, srv780957070=92, srv58568221=79, srv1674908428=36, srv1748192447=41, srv634389735=82, srv1007925750=0, srv378399700=68, srv1276636385=14, srv723854033=89, srv1579371166=32, srv688190105=87, srv798882542=93, srv850742511=95, srv1026680360=1, srv216964404=60, srv1864578648=47, srv1528136329=25, srv2049461009=55, srv224566555=61, srv1690712328=37, srv599348959=81, srv744476660=90, srv85701483=96, srv1521177070=24, srv778962024=91, srv1802523675=44, srv1655672908=34, srv1322633224=16, srv290141139=65, srv1808838765=45, srv714011844=88, srv48973133=71, srv2027164122=53, srv1786641738=43, srv380533046=69, srv1543129448=28, srv1597602939=33, srv570463511=77, srv1701897866=38, srv240139598=62, srv1066783921=3, srv579837719=78, srv595301845=80, srv1719384290=40, srv2029785947=54, srv539307596=73, srv2019399608=52, srv1376515659=20, srv177931904=42, srv1985446782=50, srv2093812916=59, srv1261948230=12, srv1216714115=10, srv12951872=15, srv64163809=83, srv127599620=13, srv496206470=72, srv385314004=70, srv1530404489=26, srv2063369666=57, srv1091030980=4, srv87442571=98, srv1383682517=21, srv1892914015=48, srv1899721301=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 03:55:30,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 03:55:30,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 03:55:30,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 03:55:30,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 03:55:30,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 03:55:30,696 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0088000612180301); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009767211922128139); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0088000612180301); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:30,696 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 03:55:30,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.640286051961868E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0088000612180301); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009767211922128139); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0088000612180301); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 03:55:31,054 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 460 ms to try 5000 different iterations.  Found a solution that moves 283 regions; Going from a computed imbalance of 6.640286051961868E-4 to a new imbalance of 2.776876743628266E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:31,055 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1108423188:83 , srv1341256932:83 , srv1376251021:83 , srv1376515659:83 , srv1719384290:83 , srv177931904:83 , srv1786641738:83 , srv270835944:83 , srv1198060599:84 , srv1261948230:84 , srv1276636385:84 , srv1576542483:84 , srv1597602939:84 , srv1690712328:84 , srv1899721301:84 , srv2027164122:84 , srv2049461009:84 , srv2063369666:84 , srv2093812916:84 , srv216964404:84 , srv290141139:84 , srv48973133:84 , srv496206470:84 , srv56978064:84 , srv579837719:84 , srv64163809:84 , srv664769527:84 , srv68751380:84 , srv723854033:84 , srv778962024:84 , srv798882542:84 , srv87442571:84 , srv1007925750:85 , srv1026680360:85 , srv1037524735:85 , srv1066783921:85 , srv1189600337:85 , srv1190747604:85 , srv1216714115:85 , srv127599620:85 , srv1322633224:85 , srv13521891:85 , srv1401871506:85 , srv140751571:85 , srv1521177070:85 , srv1528136329:85 , srv1530404489:85 , srv1538140446:85 , srv1543129448:85 , srv1547904543:85 , srv1555207078:85 , srv1579371166:85 , srv1655672908:85 , srv1674908428:85 , srv1718096565:85 , srv1808838765:85 , srv1853234732:85 , srv1892914015:85 , srv2003559001:85 , srv2019399608:85 , srv2029785947:85 , srv2071020520:85 , srv224566555:85 , srv240139598:85 , srv331359738:85 , srv352242868:85 , srv378399700:85 , srv380533046:85 , srv385314004:85 , srv539307596:85 , srv545403496:85 , srv570463511:85 , srv58568221:85 , srv595301845:85 , srv599348959:85 , srv688190105:85 , srv714011844:85 , srv744476660:85 , srv850742511:85 , srv85701483:85 , srv86290655:85 , srv915752569:85 , srv1091030980:86 , srv1183126045:86 , srv122741260:86 , srv12951872:86 , srv1664298695:86 , srv1701897866:86 , srv1748192447:86 , srv1802523675:86 , srv1864578648:86 , srv1985446782:86 , srv2050613308:86 , srv256806816:86 , srv566134612:86 , srv634389735:86 , srv661057723:86 , srv846476897:86 , srv1383682517:87 , srv780957070:607 }
2023-11-02 03:55:31,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:31,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1376251021=19, srv256806816=63, srv68751380=86, srv1190747604=8, srv1108423188=5, srv122741260=11, srv1401871506=22, srv1341256932=17, srv270835944=64, srv13521891=18, srv545403496=74, srv2071020520=58, srv1189600337=7, srv1037524735=2, srv566134612=75, srv1198060599=9, srv1547904543=29, srv1576542483=31, srv846476897=94, srv2003559001=51, srv661057723=84, srv140751571=23, srv915752569=99, srv86290655=97, srv2050613308=56, srv56978064=76, srv664769527=85, srv352242868=67, srv1853234732=46, srv331359738=66, srv1555207078=30, srv1664298695=35, srv1718096565=39, srv1538140446=27, srv1183126045=6, srv780957070=92, srv58568221=79, srv1674908428=36, srv1748192447=41, srv634389735=82, srv1007925750=0, srv378399700=68, srv1276636385=14, srv723854033=89, srv1579371166=32, srv688190105=87, srv798882542=93, srv850742511=95, srv1026680360=1, srv216964404=60, srv1864578648=47, srv1528136329=25, srv2049461009=55, srv224566555=61, srv1690712328=37, srv599348959=81, srv744476660=90, srv85701483=96, srv1521177070=24, srv778962024=91, srv1802523675=44, srv1655672908=34, srv1322633224=16, srv290141139=65, srv1808838765=45, srv714011844=88, srv48973133=71, srv2027164122=53, srv1786641738=43, srv380533046=69, srv1543129448=28, srv1597602939=33, srv570463511=77, srv1701897866=38, srv240139598=62, srv1066783921=3, srv579837719=78, srv595301845=80, srv1719384290=40, srv2029785947=54, srv539307596=73, srv2019399608=52, srv1376515659=20, srv177931904=42, srv1985446782=50, srv2093812916=59, srv1261948230=12, srv1216714115=10, srv12951872=15, srv64163809=83, srv127599620=13, srv496206470=72, srv385314004=70, srv1530404489=26, srv2063369666=57, srv1091030980=4, srv87442571=98, srv1383682517=21, srv1892914015=48, srv1899721301=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 03:55:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 03:55:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 03:55:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 03:55:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 03:55:31,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 03:55:31,177 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:31,177 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 03:55:31,178 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.578715908251806E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 03:55:31,444 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 361 ms to try 5000 different iterations.  Found a solution that moves 31 regions; Going from a computed imbalance of 2.578715908251806E-5 to a new imbalance of 6.615052311728012E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:31,445 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1189600337:83 , srv1383682517:83 , srv1538140446:83 , srv1985446782:83 , srv256806816:83 , srv331359738:83 , srv64163809:83 , srv688190105:83 , srv850742511:83 , srv1007925750:84 , srv1037524735:84 , srv1066783921:84 , srv1091030980:84 , srv1108423188:84 , srv1183126045:84 , srv1190747604:84 , srv1198060599:84 , srv1216714115:84 , srv122741260:84 , srv1261948230:84 , srv127599620:84 , srv1276636385:84 , srv12951872:84 , srv1322633224:84 , srv1341256932:84 , srv1376251021:84 , srv1401871506:84 , srv140751571:84 , srv1521177070:84 , srv1528136329:84 , srv1543129448:84 , srv1547904543:84 , srv1555207078:84 , srv1576542483:84 , srv1579371166:84 , srv1597602939:84 , srv1655672908:84 , srv1664298695:84 , srv1674908428:84 , srv1690712328:84 , srv1701897866:84 , srv1718096565:84 , srv1748192447:84 , srv177931904:84 , srv1802523675:84 , srv1808838765:84 , srv1853234732:84 , srv1864578648:84 , srv1899721301:84 , srv2003559001:84 , srv2019399608:84 , srv2027164122:84 , srv2029785947:84 , srv2049461009:84 , srv2050613308:84 , srv2063369666:84 , srv2071020520:84 , srv2093812916:84 , srv224566555:84 , srv240139598:84 , srv290141139:84 , srv352242868:84 , srv378399700:84 , srv380533046:84 , srv385314004:84 , srv48973133:84 , srv496206470:84 , srv539307596:84 , srv545403496:84 , srv566134612:84 , srv56978064:84 , srv570463511:84 , srv579837719:84 , srv58568221:84 , srv595301845:84 , srv599348959:84 , srv634389735:84 , srv661057723:84 , srv664769527:84 , srv68751380:84 , srv714011844:84 , srv723854033:84 , srv744476660:84 , srv778962024:84 , srv798882542:84 , srv846476897:84 , srv85701483:84 , srv86290655:84 , srv87442571:84 , srv915752569:84 , srv1026680360:85 , srv13521891:85 , srv1376515659:85 , srv1530404489:85 , srv1719384290:85 , srv1786641738:85 , srv1892914015:85 , srv216964404:85 , srv270835944:86 , srv780957070:683 }
2023-11-02 03:55:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1376251021=19, srv256806816=63, srv68751380=86, srv1190747604=8, srv1108423188=5, srv122741260=11, srv1401871506=22, srv1341256932=17, srv270835944=64, srv13521891=18, srv545403496=74, srv2071020520=58, srv1189600337=7, srv1037524735=2, srv566134612=75, srv1198060599=9, srv1547904543=29, srv1576542483=31, srv846476897=94, srv2003559001=51, srv661057723=84, srv140751571=23, srv915752569=99, srv86290655=97, srv2050613308=56, srv56978064=76, srv664769527=85, srv352242868=67, srv1853234732=46, srv331359738=66, srv1555207078=30, srv1664298695=35, srv1718096565=39, srv1538140446=27, srv1183126045=6, srv780957070=92, srv58568221=79, srv1674908428=36, srv1748192447=41, srv634389735=82, srv1007925750=0, srv378399700=68, srv1276636385=14, srv723854033=89, srv1579371166=32, srv688190105=87, srv798882542=93, srv850742511=95, srv1026680360=1, srv216964404=60, srv1864578648=47, srv1528136329=25, srv2049461009=55, srv224566555=61, srv1690712328=37, srv599348959=81, srv744476660=90, srv85701483=96, srv1521177070=24, srv778962024=91, srv1802523675=44, srv1655672908=34, srv1322633224=16, srv290141139=65, srv1808838765=45, srv714011844=88, srv48973133=71, srv2027164122=53, srv1786641738=43, srv380533046=69, srv1543129448=28, srv1597602939=33, srv570463511=77, srv1701897866=38, srv240139598=62, srv1066783921=3, srv579837719=78, srv595301845=80, srv1719384290=40, srv2029785947=54, srv539307596=73, srv2019399608=52, srv1376515659=20, srv177931904=42, srv1985446782=50, srv2093812916=59, srv1261948230=12, srv1216714115=10, srv12951872=15, srv64163809=83, srv127599620=13, srv496206470=72, srv385314004=70, srv1530404489=26, srv2063369666=57, srv1091030980=4, srv87442571=98, srv1383682517=21, srv1892914015=48, srv1899721301=49} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:31,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:31,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:31,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:31,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 03:55:31,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 03:55:31,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 03:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 03:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 03:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 03:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 03:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 03:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 03:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 03:55:31,566 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:31,567 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1189600337:83 , srv1383682517:83 , srv1538140446:83 , srv1985446782:83 , srv256806816:83 , srv331359738:83 , srv64163809:83 , srv688190105:83 , srv850742511:83 , srv1007925750:84 , srv1037524735:84 , srv1066783921:84 , srv1091030980:84 , srv1108423188:84 , srv1183126045:84 , srv1190747604:84 , srv1198060599:84 , srv1216714115:84 , srv122741260:84 , srv1261948230:84 , srv127599620:84 , srv1276636385:84 , srv12951872:84 , srv1322633224:84 , srv1341256932:84 , srv1376251021:84 , srv1401871506:84 , srv140751571:84 , srv1521177070:84 , srv1528136329:84 , srv1543129448:84 , srv1547904543:84 , srv1555207078:84 , srv1576542483:84 , srv1579371166:84 , srv1597602939:84 , srv1655672908:84 , srv1664298695:84 , srv1674908428:84 , srv1690712328:84 , srv1701897866:84 , srv1718096565:84 , srv1748192447:84 , srv177931904:84 , srv1802523675:84 , srv1808838765:84 , srv1853234732:84 , srv1864578648:84 , srv1899721301:84 , srv2003559001:84 , srv2019399608:84 , srv2027164122:84 , srv2029785947:84 , srv2049461009:84 , srv2050613308:84 , srv2063369666:84 , srv2071020520:84 , srv2093812916:84 , srv224566555:84 , srv240139598:84 , srv290141139:84 , srv352242868:84 , srv378399700:84 , srv380533046:84 , srv385314004:84 , srv48973133:84 , srv496206470:84 , srv539307596:84 , srv545403496:84 , srv566134612:84 , srv56978064:84 , srv570463511:84 , srv579837719:84 , srv58568221:84 , srv595301845:84 , srv599348959:84 , srv634389735:84 , srv661057723:84 , srv664769527:84 , srv68751380:84 , srv714011844:84 , srv723854033:84 , srv744476660:84 , srv778962024:84 , srv798882542:84 , srv846476897:84 , srv85701483:84 , srv86290655:84 , srv87442571:84 , srv915752569:84 , srv1026680360:85 , srv13521891:85 , srv1376515659:85 , srv1530404489:85 , srv1719384290:85 , srv1786641738:85 , srv1892914015:85 , srv216964404:85 , srv270835944:86 , srv780957070:683 }
2023-11-02 03:55:31,590 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=106 (was 89) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=11508 (was 11994)
2023-11-02 03:55:31,598 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=106, ProcessCount=167, AvailableMemoryMB=11507
2023-11-02 03:55:31,598 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-02 03:55:31,598 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-02 03:55:31,613 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1071035944:84 , srv1084711333:84 , srv1157052258:84 , srv1293503397:84 , srv1306018180:84 , srv1352077042:84 , srv1358574202:84 , srv1488947116:84 , srv1539953728:84 , srv1602440460:84 , srv1603234684:84 , srv1606954096:84 , srv1615607776:84 , srv1947456460:84 , srv2080376172:84 , srv2115740018:84 , srv331974298:84 , srv385818115:84 , srv466860393:84 , srv551130402:84 , srv638217449:84 , srv64494086:84 , srv656745902:84 , srv694225981:84 , srv764497905:84 , srv806468641:84 , srv845085019:84 , srv941501536:84 , srv973403478:84 , srv995107473:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-02 03:55:31,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:55:31,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1358574202=6, srv1084711333=1, srv1306018180=4, srv64494086=21, srv1488947116=7, srv2080376172=14, srv1539953728=8, srv385818115=17, srv331974298=16, srv973403478=28, srv1071035944=0, srv694225981=23, srv845085019=26, srv638217449=20, srv1602440460=9, srv1352077042=5, srv764497905=24, srv1606954096=11, srv1157052258=2, srv806468641=25, srv466860393=18, srv656745902=22, srv1603234684=10, srv995107473=29, srv1293503397=3, srv941501536=27, srv1947456460=13, srv2115740018=15, srv1615607776=12, srv551130402=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:55:31,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:55:31,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:55:31,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:55:31,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:55:31,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:55:31,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:55:31,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:55:31,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:55:31,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-02 03:55:31,654 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1172638054503414, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:55:31,654 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912562537686238, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1172638054503414, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-02 03:57:31,634 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5641223 different iterations.  Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912562537686238 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:57:31,640 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1071035944:90 , srv1084711333:90 , srv1157052258:90 , srv1293503397:90 , srv1306018180:90 , srv1352077042:90 , srv1358574202:90 , srv1488947116:90 , srv1539953728:90 , srv1602440460:90 , srv1603234684:90 , srv1606954096:90 , srv1615607776:90 , srv1947456460:90 , srv2080376172:90 , srv2115740018:90 , srv331974298:90 , srv385818115:90 , srv466860393:90 , srv551130402:90 , srv638217449:90 , srv64494086:90 , srv656745902:90 , srv694225981:90 , srv764497905:90 , srv806468641:90 , srv845085019:90 , srv941501536:90 , srv973403478:90 , srv995107473:90 }
2023-11-02 03:57:31,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 03:57:31,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1358574202=6, srv1084711333=1, srv1306018180=4, srv64494086=21, srv1488947116=7, srv2080376172=14, srv1539953728=8, srv385818115=17, srv331974298=16, srv973403478=28, srv1071035944=0, srv694225981=23, srv845085019=26, srv638217449=20, srv1602440460=9, srv1352077042=5, srv764497905=24, srv1606954096=11, srv1157052258=2, srv806468641=25, srv466860393=18, srv656745902=22, srv1603234684=10, srv995107473=29, srv1293503397=3, srv941501536=27, srv1947456460=13, srv2115740018=15, srv1615607776=12, srv551130402=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 03:57:31,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 03:57:31,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 03:57:31,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 03:57:31,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 03:57:31,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 03:57:31,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 03:57:31,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 03:57:31,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 03:57:31,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 03:57:31,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 03:57:31,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-02 03:57:31,657 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 03:57:31,657 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1071035944:90 , srv1084711333:90 , srv1157052258:90 , srv1293503397:90 , srv1306018180:90 , srv1352077042:90 , srv1358574202:90 , srv1488947116:90 , srv1539953728:90 , srv1602440460:90 , srv1603234684:90 , srv1606954096:90 , srv1615607776:90 , srv1947456460:90 , srv2080376172:90 , srv2115740018:90 , srv331974298:90 , srv385818115:90 , srv466860393:90 , srv551130402:90 , srv638217449:90 , srv64494086:90 , srv656745902:90 , srv694225981:90 , srv764497905:90 , srv806468641:90 , srv845085019:90 , srv941501536:90 , srv973403478:90 , srv995107473:90 }
2023-11-02 03:57:31,672 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=168 (was 106) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=11482 (was 11507)