2023-09-27 17:53:12,022 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-27 17:53:12,361 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-27 17:53:12,471 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-27 17:53:12,528 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-27 17:53:12,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-27 17:53:12,549 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=169, AvailableMemoryMB=5589 2023-09-27 17:53:12,551 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-27 17:53:12,552 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-27 17:53:12,705 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001039015:84 , srv1011911556:84 , srv103104120:84 , srv1033070134:84 , srv1051919539:84 , srv1117321733:84 , srv1139489129:84 , srv1144677678:84 , srv117414940:84 , srv1244065397:84 , srv1244622961:84 , srv1246030970:84 , srv124967287:84 , srv1251056792:84 , srv1292844627:84 , srv1300119995:84 , srv1317734661:84 , srv134214296:84 , srv1400118261:84 , srv140117759:84 , srv1413944242:84 , srv1430998307:84 , srv1431900569:84 , srv1441192583:84 , srv1454095183:84 , srv1460168241:84 , srv1484817158:84 , srv1511159740:84 , srv1545253827:84 , srv1605809379:84 , srv162423315:84 , srv1657330218:84 , srv166057846:84 , srv1675969136:84 , srv1681030975:84 , srv1687827952:84 , srv1698704621:84 , srv1699214791:84 , srv1700015672:84 , srv1721501097:84 , srv1735576449:84 , srv175041249:84 , srv1767008459:84 , srv1771033624:84 , srv1774415285:84 , srv177687283:84 , srv179040510:84 , srv1805766115:84 , srv1806158219:84 , srv1817097959:84 , srv187961578:84 , srv1896623870:84 , srv1927521085:84 , srv1987230228:84 , srv1993649690:84 , srv2003304729:84 , srv2004510667:84 , srv2022658800:84 , srv2038833869:84 , srv2049556010:84 , srv2056829095:84 , srv2059266318:84 , srv2064216082:84 , srv2070514018:84 , srv2079475995:84 , srv208029265:84 , srv2113813810:84 , srv257703094:84 , srv266184497:84 , srv294020914:84 , srv299900136:84 , srv339220229:84 , srv376340681:84 , srv379115670:84 , srv406079951:84 , srv46999773:84 , srv498761770:84 , srv511215077:84 , srv544485070:84 , srv545397488:84 , srv55120666:84 , srv559495204:84 , srv592343062:84 , srv603635822:84 , srv64122394:84 , srv655652621:84 , srv66417982:84 , srv684489727:84 , srv728910404:84 , srv737273098:84 , srv759238348:84 , srv774008962:84 , srv777091129:84 , srv838551184:84 , srv884047122:84 , srv884407833:84 , srv89780985:84 , srv944159197:84 , srv954200984:84 , srv1483345738:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-27 17:53:12,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1774415285=45, srv1605809379=30, srv1454095183=24, srv498761770=77, srv759238348=91, srv1483345738=26, srv1545253827=29, srv1431900569=22, srv1687827952=36, srv177687283=46, srv166057846=33, srv376340681=73, srv1805766115=48, srv1117321733=5, srv1698704621=37, srv944159197=98, srv1700015672=39, srv1011911556=1, srv2049556010=60, srv1927521085=53, srv1987230228=54, srv124967287=12, srv1767008459=43, srv1441192583=23, srv2038833869=59, srv774008962=92, srv257703094=68, srv64122394=85, srv1144677678=7, srv511215077=78, srv1244065397=9, srv603635822=84, srv1300119995=15, srv2064216082=63, srv2022658800=58, srv46999773=76, srv1033070134=3, srv1993649690=55, srv339220229=72, srv684489727=88, srv1657330218=32, srv294020914=70, srv884407833=96, srv1896623870=52, srv737273098=90, srv1484817158=27, srv1699214791=38, srv2003304729=56, srv1251056792=13, srv1771033624=44, srv162423315=31, srv179040510=47, srv1806158219=49, srv1460168241=25, srv66417982=87, srv1430998307=21, srv208029265=66, srv1051919539=4, srv1400118261=18, srv1817097959=50, srv1675969136=34, srv728910404=89, srv1413944242=20, srv2079475995=65, srv1246030970=11, srv266184497=69, srv1292844627=14, srv2004510667=57, srv1244622961=10, srv2070514018=64, srv175041249=42, srv1735576449=41, srv838551184=94, srv1721501097=40, srv559495204=82, srv954200984=99, srv1139489129=6, srv299900136=71, srv545397488=80, srv1681030975=35, srv134214296=17, srv544485070=79, srv55120666=81, srv140117759=19, srv1511159740=28, srv117414940=8, srv2059266318=62, srv89780985=97, srv777091129=93, srv187961578=51, srv406079951=75, srv655652621=86, srv884047122=95, srv2056829095=61, srv2113813810=67, srv592343062=83, srv1001039015=0, srv379115670=74, srv1317734661=16, srv103104120=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:53:12,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-27 17:53:12,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-27 17:53:12,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-27 17:53:12,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-27 17:53:12,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-27 17:53:12,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-27 17:53:12,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-27 17:53:12,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-27 17:53:12,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-27 17:53:12,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-27 17:53:12,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-27 17:53:12,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-27 17:53:12,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-27 17:53:12,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-27 17:53:12,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-27 17:53:12,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-27 17:53:12,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-27 17:53:12,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-27 17:53:12,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-27 17:53:13,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:13,018 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-27 17:53:13,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-27 17:53:13,385 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 552 ms to try 5000 different iterations. Found a solution that moves 4673 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1833055735178096. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037391235628261944, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05478864604212325, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5192222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037391235628261944, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17841666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24683333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:13,402 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1805766115:81 , srv603635822:81 , srv1117321733:82 , srv1774415285:82 , srv511215077:83 , srv592343062:83 , srv774008962:83 , srv777091129:83 , srv89780985:83 , srv1001039015:84 , srv117414940:84 , srv1246030970:84 , srv1317734661:84 , srv1441192583:84 , srv1484817158:84 , srv1681030975:84 , srv1699214791:84 , srv1735576449:84 , srv1771033624:84 , srv2003304729:84 , srv2056829095:84 , srv257703094:84 , srv299900136:84 , srv544485070:84 , srv1292844627:85 , srv1454095183:85 , srv1511159740:85 , srv162423315:85 , srv1700015672:85 , srv1896623870:85 , srv2004510667:85 , srv2064216082:85 , srv2113813810:85 , srv339220229:85 , srv64122394:85 , srv684489727:85 , srv737273098:85 , srv759238348:85 , srv884407833:85 , srv124967287:86 , srv134214296:86 , srv140117759:86 , srv175041249:86 , srv179040510:86 , srv2038833869:86 , srv2059266318:86 , srv266184497:86 , srv376340681:86 , srv498761770:86 , srv545397488:86 , srv838551184:86 , srv884047122:86 , srv954200984:86 , srv1011911556:87 , srv103104120:87 , srv1033070134:87 , srv1244065397:87 , srv1251056792:87 , srv1545253827:87 , srv1675969136:87 , srv1767008459:87 , srv1817097959:87 , srv208029265:87 , srv1300119995:88 , srv1430998307:88 , srv1431900569:88 , srv1605809379:88 , srv1657330218:88 , srv187961578:88 , srv1987230228:88 , srv294020914:88 , srv46999773:88 , srv655652621:88 , srv728910404:88 , srv1051919539:89 , srv1144677678:89 , srv166057846:89 , srv1687827952:89 , srv1698704621:89 , srv177687283:89 , srv2079475995:89 , srv379115670:89 , srv559495204:89 , srv66417982:89 , srv1139489129:90 , srv1721501097:90 , srv1927521085:90 , srv1993649690:90 , srv2049556010:90 , srv2070514018:90 , srv944159197:90 , srv1460168241:91 , srv2022658800:91 , srv406079951:91 , srv1244622961:92 , srv1806158219:92 , srv55120666:93 , srv1400118261:94 , srv1413944242:96 , srv1483345738:422 } 2023-09-27 17:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1774415285=45, srv1605809379=30, srv1454095183=24, srv498761770=77, srv759238348=91, srv1483345738=26, srv1545253827=29, srv1431900569=22, srv1687827952=36, srv177687283=46, srv166057846=33, srv376340681=73, srv1805766115=48, srv1117321733=5, srv1698704621=37, srv944159197=98, srv1700015672=39, srv1011911556=1, srv2049556010=60, srv1927521085=53, srv1987230228=54, srv124967287=12, srv1767008459=43, srv1441192583=23, srv2038833869=59, srv774008962=92, srv257703094=68, srv64122394=85, srv1144677678=7, srv511215077=78, srv1244065397=9, srv603635822=84, srv1300119995=15, srv2064216082=63, srv2022658800=58, srv46999773=76, srv1033070134=3, srv1993649690=55, srv339220229=72, srv684489727=88, srv1657330218=32, srv294020914=70, srv884407833=96, srv1896623870=52, srv737273098=90, srv1484817158=27, srv1699214791=38, srv2003304729=56, srv1251056792=13, srv1771033624=44, srv162423315=31, srv179040510=47, srv1806158219=49, srv1460168241=25, srv66417982=87, srv1430998307=21, srv208029265=66, srv1051919539=4, srv1400118261=18, srv1817097959=50, srv1675969136=34, srv728910404=89, srv1413944242=20, srv2079475995=65, srv1246030970=11, srv266184497=69, srv1292844627=14, srv2004510667=57, srv1244622961=10, srv2070514018=64, srv175041249=42, srv1735576449=41, srv838551184=94, srv1721501097=40, srv559495204=82, srv954200984=99, srv1139489129=6, srv299900136=71, srv545397488=80, srv1681030975=35, srv134214296=17, srv544485070=79, srv55120666=81, srv140117759=19, srv1511159740=28, srv117414940=8, srv2059266318=62, srv89780985=97, srv777091129=93, srv187961578=51, srv406079951=75, srv655652621=86, srv884047122=95, srv2056829095=61, srv2113813810=67, srv592343062=83, srv1001039015=0, srv379115670=74, srv1317734661=16, srv103104120=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:53:13,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:53:13,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:53:13,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:53:13,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:53:13,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:53:13,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:53:13,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:53:13,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-27 17:53:13,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-27 17:53:13,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-27 17:53:13,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-27 17:53:13,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-27 17:53:13,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-27 17:53:13,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-27 17:53:13,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-27 17:53:13,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-27 17:53:13,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-27 17:53:13,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-27 17:53:13,611 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037391235628261944, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05478864604212325, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037391235628261944, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17841666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24683333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:13,611 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-27 17:53:13,612 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18327285247244868, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037391235628261944, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05478864604212325, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037391235628261944, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17841666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24683333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-27 17:53:13,933 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 449 ms to try 5000 different iterations. Found a solution that moves 2595 regions; Going from a computed imbalance of 0.18327285247244868 to a new imbalance of 0.01518587443564577. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022726599316622725); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03691528810766387, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022726599316622725); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04041666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:13,948 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1400118261:77 , srv1413944242:78 , srv406079951:79 , srv166057846:80 , srv1806158219:80 , srv1051919539:81 , srv1244622961:81 , srv379115670:81 , srv55120666:81 , srv1139489129:82 , srv1721501097:82 , srv177687283:82 , srv2022658800:82 , srv2049556010:82 , srv208029265:82 , srv1011911556:83 , srv103104120:83 , srv1144677678:83 , srv1300119995:83 , srv1430998307:83 , srv1460168241:83 , srv1927521085:83 , srv1993649690:83 , srv655652621:83 , srv1431900569:84 , srv1657330218:84 , srv1698704621:84 , srv175041249:84 , srv2059266318:84 , srv2070514018:84 , srv294020914:84 , srv339220229:84 , srv559495204:84 , srv944159197:84 , srv954200984:84 , srv1033070134:85 , srv1244065397:85 , srv124967287:85 , srv134214296:85 , srv140117759:85 , srv1605809379:85 , srv1687827952:85 , srv1767008459:85 , srv179040510:85 , srv187961578:85 , srv1987230228:85 , srv2079475995:85 , srv257703094:85 , srv46999773:85 , srv498761770:85 , srv544485070:85 , srv64122394:85 , srv728910404:85 , srv884047122:85 , srv1251056792:86 , srv1292844627:86 , srv1545253827:86 , srv162423315:86 , srv1675969136:86 , srv1700015672:86 , srv2038833869:86 , srv2113813810:86 , srv266184497:86 , srv376340681:86 , srv737273098:86 , srv1001039015:87 , srv1441192583:87 , srv1511159740:87 , srv1699214791:87 , srv1896623870:87 , srv299900136:87 , srv545397488:87 , srv66417982:87 , srv684489727:87 , srv774008962:87 , srv838551184:87 , srv884407833:87 , srv1246030970:88 , srv1317734661:88 , srv1484817158:88 , srv1681030975:88 , srv1735576449:88 , srv1817097959:88 , srv2064216082:88 , srv511215077:88 , srv759238348:88 , srv117414940:89 , srv1454095183:89 , srv1771033624:89 , srv1774415285:89 , srv2004510667:89 , srv89780985:89 , srv2003304729:90 , srv2056829095:90 , srv592343062:90 , srv1117321733:91 , srv1805766115:91 , srv603635822:91 , srv777091129:91 , srv1483345738:554 } 2023-09-27 17:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1774415285=45, srv1605809379=30, srv1454095183=24, srv498761770=77, srv759238348=91, srv1483345738=26, srv1545253827=29, srv1431900569=22, srv1687827952=36, srv177687283=46, srv166057846=33, srv376340681=73, srv1805766115=48, srv1117321733=5, srv1698704621=37, srv944159197=98, srv1700015672=39, srv1011911556=1, srv2049556010=60, srv1927521085=53, srv1987230228=54, srv124967287=12, srv1767008459=43, srv1441192583=23, srv2038833869=59, srv774008962=92, srv257703094=68, srv64122394=85, srv1144677678=7, srv511215077=78, srv1244065397=9, srv603635822=84, srv1300119995=15, srv2064216082=63, srv2022658800=58, srv46999773=76, srv1033070134=3, srv1993649690=55, srv339220229=72, srv684489727=88, srv1657330218=32, srv294020914=70, srv884407833=96, srv1896623870=52, srv737273098=90, srv1484817158=27, srv1699214791=38, srv2003304729=56, srv1251056792=13, srv1771033624=44, srv162423315=31, srv179040510=47, srv1806158219=49, srv1460168241=25, srv66417982=87, srv1430998307=21, srv208029265=66, srv1051919539=4, srv1400118261=18, srv1817097959=50, srv1675969136=34, srv728910404=89, srv1413944242=20, srv2079475995=65, srv1246030970=11, srv266184497=69, srv1292844627=14, srv2004510667=57, srv1244622961=10, srv2070514018=64, srv175041249=42, srv1735576449=41, srv838551184=94, srv1721501097=40, srv559495204=82, srv954200984=99, srv1139489129=6, srv299900136=71, srv545397488=80, srv1681030975=35, srv134214296=17, srv544485070=79, srv55120666=81, srv140117759=19, srv1511159740=28, srv117414940=8, srv2059266318=62, srv89780985=97, srv777091129=93, srv187961578=51, srv406079951=75, srv655652621=86, srv884047122=95, srv2056829095=61, srv2113813810=67, srv592343062=83, srv1001039015=0, srv379115670=74, srv1317734661=16, srv103104120=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:53:13,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:53:13,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:53:13,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-27 17:53:13,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-27 17:53:13,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-27 17:53:13,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-27 17:53:13,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-27 17:53:13,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-27 17:53:13,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-27 17:53:13,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-27 17:53:13,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-27 17:53:14,157 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022726599316622725); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03691528810766387, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022726599316622725); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04041666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:14,158 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-27 17:53:14,158 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015167703857278211, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022726599316622725); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03691528810766387, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022726599316622725); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04041666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-27 17:53:14,515 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 500 ms to try 5000 different iterations. Found a solution that moves 936 regions; Going from a computed imbalance of 0.015167703857278211 to a new imbalance of 9.752253671593848E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007554102983100421); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01044465935734187); MoveCostFunction : (multiplier=7.0, imbalance=0.104, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007554102983100421); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:14,517 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1817097959:82 , srv1117321733:83 , srv1413944242:83 , srv1454095183:83 , srv2056829095:83 , srv2070514018:83 , srv55120666:83 , srv66417982:83 , srv777091129:83 , srv944159197:83 , srv1244622961:84 , srv1251056792:84 , srv1460168241:84 , srv1687827952:84 , srv1698704621:84 , srv1771033624:84 , srv187961578:84 , srv1927521085:84 , srv1993649690:84 , srv2004510667:84 , srv2022658800:84 , srv2064216082:84 , srv2079475995:84 , srv545397488:84 , srv728910404:84 , srv838551184:84 , srv89780985:84 , srv1033070134:85 , srv1139489129:85 , srv1144677678:85 , srv1246030970:85 , srv1400118261:85 , srv1511159740:85 , srv1545253827:85 , srv1605809379:85 , srv1657330218:85 , srv1675969136:85 , srv1721501097:85 , srv1806158219:85 , srv1896623870:85 , srv1987230228:85 , srv2003304729:85 , srv266184497:85 , srv294020914:85 , srv376340681:85 , srv46999773:85 , srv559495204:85 , srv592343062:85 , srv603635822:85 , srv684489727:85 , srv759238348:85 , srv884407833:85 , srv117414940:86 , srv1244065397:86 , srv124967287:86 , srv1292844627:86 , srv1317734661:86 , srv134214296:86 , srv140117759:86 , srv1430998307:86 , srv1431900569:86 , srv1441192583:86 , srv1484817158:86 , srv162423315:86 , srv1681030975:86 , srv1699214791:86 , srv1735576449:86 , srv1767008459:86 , srv177687283:86 , srv179040510:86 , srv1805766115:86 , srv2038833869:86 , srv2049556010:86 , srv299900136:86 , srv379115670:86 , srv498761770:86 , srv511215077:86 , srv655652621:86 , srv737273098:86 , srv884047122:86 , srv1001039015:87 , srv1011911556:87 , srv1300119995:87 , srv1700015672:87 , srv1774415285:87 , srv2059266318:87 , srv208029265:87 , srv257703094:87 , srv406079951:87 , srv64122394:87 , srv774008962:87 , srv954200984:87 , srv103104120:88 , srv1051919539:88 , srv166057846:88 , srv175041249:88 , srv2113813810:88 , srv339220229:88 , srv544485070:89 , srv1483345738:549 } 2023-09-27 17:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1774415285=45, srv1605809379=30, srv1454095183=24, srv498761770=77, srv759238348=91, srv1483345738=26, srv1545253827=29, srv1431900569=22, srv1687827952=36, srv177687283=46, srv166057846=33, srv376340681=73, srv1805766115=48, srv1117321733=5, srv1698704621=37, srv944159197=98, srv1700015672=39, srv1011911556=1, srv2049556010=60, srv1927521085=53, srv1987230228=54, srv124967287=12, srv1767008459=43, srv1441192583=23, srv2038833869=59, srv774008962=92, srv257703094=68, srv64122394=85, srv1144677678=7, srv511215077=78, srv1244065397=9, srv603635822=84, srv1300119995=15, srv2064216082=63, srv2022658800=58, srv46999773=76, srv1033070134=3, srv1993649690=55, srv339220229=72, srv684489727=88, srv1657330218=32, srv294020914=70, srv884407833=96, srv1896623870=52, srv737273098=90, srv1484817158=27, srv1699214791=38, srv2003304729=56, srv1251056792=13, srv1771033624=44, srv162423315=31, srv179040510=47, srv1806158219=49, srv1460168241=25, srv66417982=87, srv1430998307=21, srv208029265=66, srv1051919539=4, srv1400118261=18, srv1817097959=50, srv1675969136=34, srv728910404=89, srv1413944242=20, srv2079475995=65, srv1246030970=11, srv266184497=69, srv1292844627=14, srv2004510667=57, srv1244622961=10, srv2070514018=64, srv175041249=42, srv1735576449=41, srv838551184=94, srv1721501097=40, srv559495204=82, srv954200984=99, srv1139489129=6, srv299900136=71, srv545397488=80, srv1681030975=35, srv134214296=17, srv544485070=79, srv55120666=81, srv140117759=19, srv1511159740=28, srv117414940=8, srv2059266318=62, srv89780985=97, srv777091129=93, srv187961578=51, srv406079951=75, srv655652621=86, srv884047122=95, srv2056829095=61, srv2113813810=67, srv592343062=83, srv1001039015=0, srv379115670=74, srv1317734661=16, srv103104120=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:53:14,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:53:14,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:53:14,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:53:14,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:53:14,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-27 17:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-27 17:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-27 17:53:14,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-27 17:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-27 17:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-27 17:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-27 17:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-27 17:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-27 17:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-27 17:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-27 17:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-27 17:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-27 17:53:14,726 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007554102983100421); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01044465935734187); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007554102983100421); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:14,726 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-27 17:53:14,727 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.686713550776758E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007554102983100421); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01044465935734187); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007554102983100421); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-27 17:53:15,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 481 ms to try 5000 different iterations. Found a solution that moves 269 regions; Going from a computed imbalance of 9.686713550776758E-4 to a new imbalance of 1.9055587189032026E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.02988888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:15,060 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv103104120:83 , srv117414940:83 , srv175041249:83 , srv2003304729:83 , srv2113813810:83 , srv1001039015:84 , srv1244065397:84 , srv1300119995:84 , srv1317734661:84 , srv1431900569:84 , srv1460168241:84 , srv1484817158:84 , srv1545253827:84 , srv1605809379:84 , srv1675969136:84 , srv1681030975:84 , srv1687827952:84 , srv1699214791:84 , srv1700015672:84 , srv1735576449:84 , srv1767008459:84 , srv1774415285:84 , srv1805766115:84 , srv1896623870:84 , srv1987230228:84 , srv2004510667:84 , srv2022658800:84 , srv2038833869:84 , srv2049556010:84 , srv2079475995:84 , srv511215077:84 , srv544485070:84 , srv559495204:84 , srv655652621:84 , srv66417982:84 , srv1011911556:85 , srv1033070134:85 , srv1051919539:85 , srv1117321733:85 , srv1139489129:85 , srv1144677678:85 , srv1244622961:85 , srv1246030970:85 , srv124967287:85 , srv1251056792:85 , srv1292844627:85 , srv134214296:85 , srv140117759:85 , srv1413944242:85 , srv1441192583:85 , srv1511159740:85 , srv162423315:85 , srv1657330218:85 , srv166057846:85 , srv1721501097:85 , srv1771033624:85 , srv177687283:85 , srv179040510:85 , srv1806158219:85 , srv1817097959:85 , srv187961578:85 , srv1927521085:85 , srv1993649690:85 , srv2056829095:85 , srv2059266318:85 , srv2064216082:85 , srv2070514018:85 , srv208029265:85 , srv266184497:85 , srv294020914:85 , srv339220229:85 , srv376340681:85 , srv406079951:85 , srv46999773:85 , srv498761770:85 , srv545397488:85 , srv55120666:85 , srv592343062:85 , srv603635822:85 , srv64122394:85 , srv684489727:85 , srv728910404:85 , srv737273098:85 , srv759238348:85 , srv774008962:85 , srv777091129:85 , srv838551184:85 , srv884047122:85 , srv884407833:85 , srv944159197:85 , srv954200984:85 , srv1400118261:86 , srv1430998307:86 , srv1454095183:86 , srv1698704621:86 , srv257703094:86 , srv299900136:86 , srv379115670:86 , srv89780985:86 , srv1483345738:617 } 2023-09-27 17:53:15,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:53:15,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1774415285=45, srv1605809379=30, srv1454095183=24, srv498761770=77, srv759238348=91, srv1483345738=26, srv1545253827=29, srv1431900569=22, srv1687827952=36, srv177687283=46, srv166057846=33, srv376340681=73, srv1805766115=48, srv1117321733=5, srv1698704621=37, srv944159197=98, srv1700015672=39, srv1011911556=1, srv2049556010=60, srv1927521085=53, srv1987230228=54, srv124967287=12, srv1767008459=43, srv1441192583=23, srv2038833869=59, srv774008962=92, srv257703094=68, srv64122394=85, srv1144677678=7, srv511215077=78, srv1244065397=9, srv603635822=84, srv1300119995=15, srv2064216082=63, srv2022658800=58, srv46999773=76, srv1033070134=3, srv1993649690=55, srv339220229=72, srv684489727=88, srv1657330218=32, srv294020914=70, srv884407833=96, srv1896623870=52, srv737273098=90, srv1484817158=27, srv1699214791=38, srv2003304729=56, srv1251056792=13, srv1771033624=44, srv162423315=31, srv179040510=47, srv1806158219=49, srv1460168241=25, srv66417982=87, srv1430998307=21, srv208029265=66, srv1051919539=4, srv1400118261=18, srv1817097959=50, srv1675969136=34, srv728910404=89, srv1413944242=20, srv2079475995=65, srv1246030970=11, srv266184497=69, srv1292844627=14, srv2004510667=57, srv1244622961=10, srv2070514018=64, srv175041249=42, srv1735576449=41, srv838551184=94, srv1721501097=40, srv559495204=82, srv954200984=99, srv1139489129=6, srv299900136=71, srv545397488=80, srv1681030975=35, srv134214296=17, srv544485070=79, srv55120666=81, srv140117759=19, srv1511159740=28, srv117414940=8, srv2059266318=62, srv89780985=97, srv777091129=93, srv187961578=51, srv406079951=75, srv655652621=86, srv884047122=95, srv2056829095=61, srv2113813810=67, srv592343062=83, srv1001039015=0, srv379115670=74, srv1317734661=16, srv103104120=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:53:15,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:53:15,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-27 17:53:15,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-27 17:53:15,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-27 17:53:15,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-27 17:53:15,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-27 17:53:15,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-27 17:53:15,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-27 17:53:15,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-27 17:53:15,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-27 17:53:15,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-27 17:53:15,187 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7172008930506658E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:15,187 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv103104120:83 , srv117414940:83 , srv175041249:83 , srv2003304729:83 , srv2113813810:83 , srv1001039015:84 , srv1244065397:84 , srv1300119995:84 , srv1317734661:84 , srv1431900569:84 , srv1460168241:84 , srv1484817158:84 , srv1545253827:84 , srv1605809379:84 , srv1675969136:84 , srv1681030975:84 , srv1687827952:84 , srv1699214791:84 , srv1700015672:84 , srv1735576449:84 , srv1767008459:84 , srv1774415285:84 , srv1805766115:84 , srv1896623870:84 , srv1987230228:84 , srv2004510667:84 , srv2022658800:84 , srv2038833869:84 , srv2049556010:84 , srv2079475995:84 , srv511215077:84 , srv544485070:84 , srv559495204:84 , srv655652621:84 , srv66417982:84 , srv1011911556:85 , srv1033070134:85 , srv1051919539:85 , srv1117321733:85 , srv1139489129:85 , srv1144677678:85 , srv1244622961:85 , srv1246030970:85 , srv124967287:85 , srv1251056792:85 , srv1292844627:85 , srv134214296:85 , srv140117759:85 , srv1413944242:85 , srv1441192583:85 , srv1511159740:85 , srv162423315:85 , srv1657330218:85 , srv166057846:85 , srv1721501097:85 , srv1771033624:85 , srv177687283:85 , srv179040510:85 , srv1806158219:85 , srv1817097959:85 , srv187961578:85 , srv1927521085:85 , srv1993649690:85 , srv2056829095:85 , srv2059266318:85 , srv2064216082:85 , srv2070514018:85 , srv208029265:85 , srv266184497:85 , srv294020914:85 , srv339220229:85 , srv376340681:85 , srv406079951:85 , srv46999773:85 , srv498761770:85 , srv545397488:85 , srv55120666:85 , srv592343062:85 , srv603635822:85 , srv64122394:85 , srv684489727:85 , srv728910404:85 , srv737273098:85 , srv759238348:85 , srv774008962:85 , srv777091129:85 , srv838551184:85 , srv884047122:85 , srv884407833:85 , srv944159197:85 , srv954200984:85 , srv1400118261:86 , srv1430998307:86 , srv1454095183:86 , srv1698704621:86 , srv257703094:86 , srv299900136:86 , srv379115670:86 , srv89780985:86 , srv1483345738:617 } 2023-09-27 17:53:15,216 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=271 (was 260) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4965 (was 5589) 2023-09-27 17:53:15,225 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=169, AvailableMemoryMB=4964 2023-09-27 17:53:15,226 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-27 17:53:15,226 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-27 17:53:15,234 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1061097714:84 , srv1065653041:84 , srv1070333350:84 , srv1127235330:84 , srv1354826179:84 , srv1379725330:84 , srv1443349005:84 , srv1486714376:84 , srv1640705847:84 , srv1687751008:84 , srv1774929768:84 , srv1850570733:84 , srv1909790974:84 , srv1930828280:84 , srv1977839488:84 , srv2008096096:84 , srv2063754037:84 , srv2103231596:84 , srv279733526:84 , srv33378935:84 , srv363625658:84 , srv512785203:84 , srv654942217:84 , srv675061251:84 , srv773469905:84 , srv82883735:84 , srv894455747:84 , srv962873323:84 , srv976523325:84 , srv1267654238:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-27 17:53:15,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:53:15,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33378935=20, srv1850570733=12, srv512785203=22, srv82883735=26, srv1909790974=13, srv1354826179=5, srv1687751008=10, srv2008096096=16, srv1267654238=4, srv2103231596=18, srv1774929768=11, srv1486714376=8, srv962873323=28, srv773469905=25, srv363625658=21, srv1061097714=0, srv1443349005=7, srv1977839488=15, srv1127235330=3, srv675061251=24, srv1070333350=2, srv1930828280=14, srv654942217=23, srv1640705847=9, srv976523325=29, srv1065653041=1, srv2063754037=17, srv1379725330=6, srv894455747=27, srv279733526=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:53:15,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:53:15,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-27 17:53:15,260 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0980382909764035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:53:15,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912501958717902, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0980382909764035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-27 17:55:15,248 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5889048 different iterations. Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912501958717902 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:55:15,254 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1061097714:90 , srv1065653041:90 , srv1070333350:90 , srv1127235330:90 , srv1267654238:90 , srv1354826179:90 , srv1379725330:90 , srv1443349005:90 , srv1486714376:90 , srv1640705847:90 , srv1687751008:90 , srv1774929768:90 , srv1850570733:90 , srv1909790974:90 , srv1930828280:90 , srv1977839488:90 , srv2008096096:90 , srv2063754037:90 , srv2103231596:90 , srv279733526:90 , srv33378935:90 , srv363625658:90 , srv512785203:90 , srv654942217:90 , srv675061251:90 , srv773469905:90 , srv82883735:90 , srv894455747:90 , srv962873323:90 , srv976523325:90 } 2023-09-27 17:55:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-27 17:55:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33378935=20, srv1850570733=12, srv512785203=22, srv82883735=26, srv1909790974=13, srv1354826179=5, srv1687751008=10, srv2008096096=16, srv1267654238=4, srv2103231596=18, srv1774929768=11, srv1486714376=8, srv962873323=28, srv773469905=25, srv363625658=21, srv1061097714=0, srv1443349005=7, srv1977839488=15, srv1127235330=3, srv675061251=24, srv1070333350=2, srv1930828280=14, srv654942217=23, srv1640705847=9, srv976523325=29, srv1065653041=1, srv2063754037=17, srv1379725330=6, srv894455747=27, srv279733526=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-27 17:55:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-27 17:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-27 17:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-27 17:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-27 17:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-27 17:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-27 17:55:15,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-27 17:55:15,272 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1061097714:90 , srv1065653041:90 , srv1070333350:90 , srv1127235330:90 , srv1267654238:90 , srv1354826179:90 , srv1379725330:90 , srv1443349005:90 , srv1486714376:90 , srv1640705847:90 , srv1687751008:90 , srv1774929768:90 , srv1850570733:90 , srv1909790974:90 , srv1930828280:90 , srv1977839488:90 , srv2008096096:90 , srv2063754037:90 , srv2103231596:90 , srv279733526:90 , srv33378935:90 , srv363625658:90 , srv512785203:90 , srv654942217:90 , srv675061251:90 , srv773469905:90 , srv82883735:90 , srv894455747:90 , srv962873323:90 , srv976523325:90 } 2023-09-27 17:55:15,286 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 271), ProcessCount=166 (was 169), AvailableMemoryMB=3786 (was 4964)