2023-05-15 10:53:22,819 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-15 10:53:23,145 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-15 10:53:23,265 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-15 10:53:23,338 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 10:53:23,347 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-15 10:53:23,360 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=171, AvailableMemoryMB=14905 2023-05-15 10:53:23,362 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 10:53:23,363 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-15 10:53:23,517 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000583384:84 , srv106001744:84 , srv1090407986:84 , srv1109034982:84 , srv1111616716:84 , srv1119872601:84 , srv1143835153:84 , srv1149975422:84 , srv1192457820:84 , srv1202111045:84 , srv1259441205:84 , srv127252726:84 , srv1306672778:84 , srv1311828717:84 , srv1319757216:84 , srv1353945449:84 , srv1367980824:84 , srv1445802196:84 , srv1457326962:84 , srv1493646711:84 , srv1495277275:84 , srv1516273043:84 , srv151838048:84 , srv1596792077:84 , srv1680398819:84 , srv1709138241:84 , srv1709244878:84 , srv1722504743:84 , srv1768401386:84 , srv1779078570:84 , srv1815477317:84 , srv182477840:84 , srv1834253353:84 , srv187795944:84 , srv1884848869:84 , srv1893576895:84 , srv1900187548:84 , srv192133641:84 , srv1958983033:84 , srv1992810572:84 , srv2008299073:84 , srv2021654246:84 , srv2075022627:84 , srv2085478635:84 , srv2108477493:84 , srv2133899851:84 , srv2138354312:84 , srv214613420:84 , srv215070013:84 , srv236128661:84 , srv239216161:84 , srv241454287:84 , srv264453119:84 , srv266743310:84 , srv299944746:84 , srv310914627:84 , srv327523118:84 , srv332028200:84 , srv343381741:84 , srv345801564:84 , srv345831738:84 , srv348615924:84 , srv367142132:84 , srv374041468:84 , srv390654907:84 , srv391227267:84 , srv397007354:84 , srv400706978:84 , srv40076264:84 , srv433850538:84 , srv454787009:84 , srv477272793:84 , srv50798747:84 , srv527240190:84 , srv529267796:84 , srv551252176:84 , srv581949915:84 , srv610113787:84 , srv633228506:84 , srv656535658:84 , srv700253227:84 , srv700745774:84 , srv702983858:84 , srv704525476:84 , srv745612046:84 , srv761932927:84 , srv802162159:84 , srv817445393:84 , srv833742181:84 , srv871668953:84 , srv919157665:84 , srv959781299:84 , srv964035624:84 , srv966163484:84 , srv968099843:84 , srv968263924:84 , srv981034998:84 , srv988138455:84 , srv997271246:84 , srv2017644481:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-15 10:53:23,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv988138455=98, srv1893576895=35, srv1090407986=2, srv2075022627=43, srv2133899851=46, srv266743310=54, srv1779078570=29, srv581949915=77, srv332028200=58, srv1306672778=12, srv182477840=31, srv610113787=78, srv700745774=82, srv327523118=57, srv1493646711=19, srv50798747=73, srv529267796=75, srv968263924=96, srv700253227=81, srv997271246=99, srv310914627=56, srv214613420=48, srv1900187548=36, srv367142132=63, srv1367980824=16, srv400706978=68, srv1259441205=10, srv802162159=87, srv374041468=64, srv964035624=93, srv2138354312=47, srv477272793=72, srv215070013=49, srv397007354=67, srv1457326962=18, srv633228506=79, srv1596792077=23, srv345801564=60, srv299944746=55, srv2017644481=41, srv151838048=22, srv239216161=51, srv761932927=86, srv1680398819=24, srv551252176=76, srv968099843=95, srv2108477493=45, srv1311828717=13, srv192133641=37, srv704525476=84, srv959781299=92, srv343381741=59, srv1815477317=30, srv1000583384=0, srv433850538=70, srv1516273043=21, srv1709138241=25, srv40076264=69, srv981034998=97, srv1834253353=32, srv345831738=61, srv1495277275=20, srv1445802196=17, srv833742181=89, srv1709244878=26, srv454787009=71, srv1109034982=3, srv1149975422=7, srv187795944=33, srv871668953=90, srv241454287=52, srv1353945449=15, srv1722504743=27, srv1202111045=9, srv264453119=53, srv106001744=1, srv1768401386=28, srv2008299073=40, srv817445393=88, srv1958983033=38, srv2085478635=44, srv1143835153=6, srv1319757216=14, srv391227267=66, srv745612046=85, srv1119872601=5, srv656535658=80, srv236128661=50, srv527240190=74, srv702983858=83, srv966163484=94, srv390654907=65, srv1884848869=34, srv348615924=62, srv2021654246=42, srv1992810572=39, srv1192457820=8, srv127252726=11, srv919157665=91, srv1111616716=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:23,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:23,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:23,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:23,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:23,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:23,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:23,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:23,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:23,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:23,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:23,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:23,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:23,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:23,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:23,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:23,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:23,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:23,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:23,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 10:53:23,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 10:53:23,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 10:53:23,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 10:53:23,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 10:53:23,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 10:53:23,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 10:53:23,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 10:53:23,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 10:53:23,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 10:53:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 10:53:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 10:53:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 10:53:23,928 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:23,929 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 10:53:23,929 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 10:53:24,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 682 ms to try 5000 different iterations. Found a solution that moves 4707 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17792384967328514. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039506558640091526, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05690088257409052, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.523, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039506558640091526, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:24,353 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv127252726:79 , srv1722504743:79 , srv1596792077:80 , srv1111616716:82 , srv1353945449:82 , srv1768401386:82 , srv1992810572:83 , srv266743310:83 , srv817445393:83 , srv833742181:83 , srv959781299:83 , srv1306672778:84 , srv151838048:84 , srv2075022627:84 , srv236128661:84 , srv327523118:84 , srv633228506:84 , srv1192457820:85 , srv1259441205:85 , srv1493646711:85 , srv1709244878:85 , srv1900187548:85 , srv192133641:85 , srv214613420:85 , srv239216161:85 , srv400706978:85 , srv551252176:85 , srv656535658:85 , srv745612046:85 , srv919157665:85 , srv997271246:85 , srv1119872601:86 , srv1149975422:86 , srv1319757216:86 , srv1457326962:86 , srv1516273043:86 , srv1709138241:86 , srv182477840:86 , srv1834253353:86 , srv1893576895:86 , srv1958983033:86 , srv2108477493:86 , srv2133899851:86 , srv2138354312:86 , srv310914627:86 , srv367142132:86 , srv390654907:86 , srv391227267:86 , srv964035624:86 , srv968099843:86 , srv1109034982:87 , srv1367980824:87 , srv1445802196:87 , srv1495277275:87 , srv1815477317:87 , srv215070013:87 , srv299944746:87 , srv343381741:87 , srv345831738:87 , srv40076264:87 , srv433850538:87 , srv454787009:87 , srv50798747:87 , srv610113787:87 , srv700745774:87 , srv702983858:87 , srv704525476:87 , srv1143835153:88 , srv1680398819:88 , srv1779078570:88 , srv1884848869:88 , srv2021654246:88 , srv241454287:88 , srv332028200:88 , srv345801564:88 , srv477272793:88 , srv527240190:88 , srv529267796:88 , srv581949915:88 , srv761932927:88 , srv802162159:88 , srv871668953:88 , srv968263924:88 , srv988138455:88 , srv106001744:89 , srv1090407986:89 , srv1311828717:89 , srv2008299073:89 , srv2085478635:89 , srv264453119:89 , srv397007354:89 , srv700253227:89 , srv966163484:89 , srv348615924:90 , srv374041468:90 , srv981034998:91 , srv187795944:94 , srv1000583384:95 , srv1202111045:96 , srv2017644481:441 } 2023-05-15 10:53:24,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:24,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv988138455=98, srv1893576895=35, srv1090407986=2, srv2075022627=43, srv2133899851=46, srv266743310=54, srv1779078570=29, srv581949915=77, srv332028200=58, srv1306672778=12, srv182477840=31, srv610113787=78, srv700745774=82, srv327523118=57, srv1493646711=19, srv50798747=73, srv529267796=75, srv968263924=96, srv700253227=81, srv997271246=99, srv310914627=56, srv214613420=48, srv1900187548=36, srv367142132=63, srv1367980824=16, srv400706978=68, srv1259441205=10, srv802162159=87, srv374041468=64, srv964035624=93, srv2138354312=47, srv477272793=72, srv215070013=49, srv397007354=67, srv1457326962=18, srv633228506=79, srv1596792077=23, srv345801564=60, srv299944746=55, srv2017644481=41, srv151838048=22, srv239216161=51, srv761932927=86, srv1680398819=24, srv551252176=76, srv968099843=95, srv2108477493=45, srv1311828717=13, srv192133641=37, srv704525476=84, srv959781299=92, srv343381741=59, srv1815477317=30, srv1000583384=0, srv433850538=70, srv1516273043=21, srv1709138241=25, srv40076264=69, srv981034998=97, srv1834253353=32, srv345831738=61, srv1495277275=20, srv1445802196=17, srv833742181=89, srv1709244878=26, srv454787009=71, srv1109034982=3, srv1149975422=7, srv187795944=33, srv871668953=90, srv241454287=52, srv1353945449=15, srv1722504743=27, srv1202111045=9, srv264453119=53, srv106001744=1, srv1768401386=28, srv2008299073=40, srv817445393=88, srv1958983033=38, srv2085478635=44, srv1143835153=6, srv1319757216=14, srv391227267=66, srv745612046=85, srv1119872601=5, srv656535658=80, srv236128661=50, srv527240190=74, srv702983858=83, srv966163484=94, srv390654907=65, srv1884848869=34, srv348615924=62, srv2021654246=42, srv1992810572=39, srv1192457820=8, srv127252726=11, srv919157665=91, srv1111616716=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:24,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:24,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:24,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:24,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:24,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:24,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:24,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 10:53:24,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 10:53:24,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 10:53:24,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 10:53:24,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 10:53:24,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 10:53:24,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 10:53:24,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 10:53:24,596 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039506558640091526, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05690088257409052, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039506558640091526, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:24,597 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 10:53:24,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17789089055483578, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039506558640091526, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05690088257409052, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039506558640091526, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 10:53:24,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 561 ms to try 5000 different iterations. Found a solution that moves 2646 regions; Going from a computed imbalance of 0.17789089055483578 to a new imbalance of 0.014766270116304991. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023187239719212674); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034792958251394446, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.294, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023187239719212674); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:24,991 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv187795944:76 , srv1000583384:77 , srv1202111045:77 , srv1311828717:81 , srv2085478635:81 , srv1680398819:82 , srv1779078570:82 , srv1884848869:82 , srv374041468:82 , srv397007354:82 , srv106001744:83 , srv1090407986:83 , srv1143835153:83 , srv2008299073:83 , srv241454287:83 , srv264453119:83 , srv348615924:83 , srv477272793:83 , srv581949915:83 , srv700745774:83 , srv761932927:83 , srv966163484:83 , srv1109034982:84 , srv1119872601:84 , srv1834253353:84 , srv1958983033:84 , srv2021654246:84 , srv2133899851:84 , srv299944746:84 , srv391227267:84 , srv40076264:84 , srv433850538:84 , srv50798747:84 , srv529267796:84 , srv610113787:84 , srv700253227:84 , srv704525476:84 , srv802162159:84 , srv968099843:84 , srv968263924:84 , srv1149975422:85 , srv1259441205:85 , srv1367980824:85 , srv1445802196:85 , srv1457326962:85 , srv1493646711:85 , srv1516273043:85 , srv1709138241:85 , srv1815477317:85 , srv1893576895:85 , srv2108477493:85 , srv332028200:85 , srv343381741:85 , srv345801564:85 , srv390654907:85 , srv551252176:85 , srv871668953:85 , srv988138455:85 , srv1306672778:86 , srv1319757216:86 , srv1495277275:86 , srv367142132:86 , srv454787009:86 , srv527240190:86 , srv656535658:86 , srv702983858:86 , srv1192457820:87 , srv151838048:87 , srv1709244878:87 , srv182477840:87 , srv192133641:87 , srv215070013:87 , srv345831738:87 , srv919157665:87 , srv964035624:87 , srv1900187548:88 , srv2075022627:88 , srv236128661:88 , srv310914627:88 , srv327523118:88 , srv400706978:88 , srv633228506:88 , srv1992810572:89 , srv2138354312:89 , srv214613420:89 , srv239216161:89 , srv266743310:89 , srv959781299:89 , srv981034998:89 , srv997271246:89 , srv1111616716:90 , srv1353945449:90 , srv817445393:90 , srv1596792077:91 , srv1768401386:91 , srv833742181:91 , srv1722504743:92 , srv745612046:93 , srv127252726:94 , srv2017644481:539 } 2023-05-15 10:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv988138455=98, srv1893576895=35, srv1090407986=2, srv2075022627=43, srv2133899851=46, srv266743310=54, srv1779078570=29, srv581949915=77, srv332028200=58, srv1306672778=12, srv182477840=31, srv610113787=78, srv700745774=82, srv327523118=57, srv1493646711=19, srv50798747=73, srv529267796=75, srv968263924=96, srv700253227=81, srv997271246=99, srv310914627=56, srv214613420=48, srv1900187548=36, srv367142132=63, srv1367980824=16, srv400706978=68, srv1259441205=10, srv802162159=87, srv374041468=64, srv964035624=93, srv2138354312=47, srv477272793=72, srv215070013=49, srv397007354=67, srv1457326962=18, srv633228506=79, srv1596792077=23, srv345801564=60, srv299944746=55, srv2017644481=41, srv151838048=22, srv239216161=51, srv761932927=86, srv1680398819=24, srv551252176=76, srv968099843=95, srv2108477493=45, srv1311828717=13, srv192133641=37, srv704525476=84, srv959781299=92, srv343381741=59, srv1815477317=30, srv1000583384=0, srv433850538=70, srv1516273043=21, srv1709138241=25, srv40076264=69, srv981034998=97, srv1834253353=32, srv345831738=61, srv1495277275=20, srv1445802196=17, srv833742181=89, srv1709244878=26, srv454787009=71, srv1109034982=3, srv1149975422=7, srv187795944=33, srv871668953=90, srv241454287=52, srv1353945449=15, srv1722504743=27, srv1202111045=9, srv264453119=53, srv106001744=1, srv1768401386=28, srv2008299073=40, srv817445393=88, srv1958983033=38, srv2085478635=44, srv1143835153=6, srv1319757216=14, srv391227267=66, srv745612046=85, srv1119872601=5, srv656535658=80, srv236128661=50, srv527240190=74, srv702983858=83, srv966163484=94, srv390654907=65, srv1884848869=34, srv348615924=62, srv2021654246=42, srv1992810572=39, srv1192457820=8, srv127252726=11, srv919157665=91, srv1111616716=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:25,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:25,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:25,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 10:53:25,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 10:53:25,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 10:53:25,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 10:53:25,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 10:53:25,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 10:53:25,147 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023187239719212674); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034792958251394446, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023187239719212674); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:25,147 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 10:53:25,147 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014747742428304775, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023187239719212674); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034792958251394446, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023187239719212674); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 10:53:25,441 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 399 ms to try 5000 different iterations. Found a solution that moves 980 regions; Going from a computed imbalance of 0.014747742428304775 to a new imbalance of 8.224397083218021E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0074477018519587814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009881452203147957); MoveCostFunction : (multiplier=7.0, imbalance=0.10888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0074477018519587814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:25,445 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv981034998:77 , srv745612046:81 , srv345831738:82 , srv182477840:83 , srv2138354312:83 , srv214613420:83 , srv239216161:83 , srv1202111045:84 , srv127252726:84 , srv1495277275:84 , srv1768401386:84 , srv1900187548:84 , srv192133641:84 , srv1992810572:84 , srv215070013:84 , srv266743310:84 , srv310914627:84 , srv332028200:84 , srv345801564:84 , srv348615924:84 , srv400706978:84 , srv454787009:84 , srv527240190:84 , srv700253227:84 , srv702983858:84 , srv817445393:84 , srv833742181:84 , srv997271246:84 , srv1000583384:85 , srv106001744:85 , srv1111616716:85 , srv1192457820:85 , srv1367980824:85 , srv1457326962:85 , srv1815477317:85 , srv2008299073:85 , srv2075022627:85 , srv327523118:85 , srv374041468:85 , srv477272793:85 , srv529267796:85 , srv633228506:85 , srv871668953:85 , srv959781299:85 , srv964035624:85 , srv968263924:85 , srv988138455:85 , srv1090407986:86 , srv1109034982:86 , srv1319757216:86 , srv1353945449:86 , srv1445802196:86 , srv1516273043:86 , srv151838048:86 , srv1596792077:86 , srv1709244878:86 , srv2021654246:86 , srv2108477493:86 , srv2133899851:86 , srv236128661:86 , srv241454287:86 , srv264453119:86 , srv299944746:86 , srv343381741:86 , srv367142132:86 , srv390654907:86 , srv40076264:86 , srv433850538:86 , srv50798747:86 , srv551252176:86 , srv610113787:86 , srv656535658:86 , srv802162159:86 , srv919157665:86 , srv966163484:86 , srv1119872601:87 , srv1149975422:87 , srv1306672778:87 , srv1311828717:87 , srv1680398819:87 , srv1709138241:87 , srv1722504743:87 , srv1779078570:87 , srv1834253353:87 , srv187795944:87 , srv1884848869:87 , srv1893576895:87 , srv581949915:87 , srv761932927:87 , srv968099843:87 , srv1143835153:88 , srv1259441205:88 , srv1493646711:88 , srv1958983033:88 , srv2085478635:88 , srv391227267:88 , srv397007354:88 , srv700745774:88 , srv704525476:88 , srv2017644481:544 } 2023-05-15 10:53:25,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:25,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv988138455=98, srv1893576895=35, srv1090407986=2, srv2075022627=43, srv2133899851=46, srv266743310=54, srv1779078570=29, srv581949915=77, srv332028200=58, srv1306672778=12, srv182477840=31, srv610113787=78, srv700745774=82, srv327523118=57, srv1493646711=19, srv50798747=73, srv529267796=75, srv968263924=96, srv700253227=81, srv997271246=99, srv310914627=56, srv214613420=48, srv1900187548=36, srv367142132=63, srv1367980824=16, srv400706978=68, srv1259441205=10, srv802162159=87, srv374041468=64, srv964035624=93, srv2138354312=47, srv477272793=72, srv215070013=49, srv397007354=67, srv1457326962=18, srv633228506=79, srv1596792077=23, srv345801564=60, srv299944746=55, srv2017644481=41, srv151838048=22, srv239216161=51, srv761932927=86, srv1680398819=24, srv551252176=76, srv968099843=95, srv2108477493=45, srv1311828717=13, srv192133641=37, srv704525476=84, srv959781299=92, srv343381741=59, srv1815477317=30, srv1000583384=0, srv433850538=70, srv1516273043=21, srv1709138241=25, srv40076264=69, srv981034998=97, srv1834253353=32, srv345831738=61, srv1495277275=20, srv1445802196=17, srv833742181=89, srv1709244878=26, srv454787009=71, srv1109034982=3, srv1149975422=7, srv187795944=33, srv871668953=90, srv241454287=52, srv1353945449=15, srv1722504743=27, srv1202111045=9, srv264453119=53, srv106001744=1, srv1768401386=28, srv2008299073=40, srv817445393=88, srv1958983033=38, srv2085478635=44, srv1143835153=6, srv1319757216=14, srv391227267=66, srv745612046=85, srv1119872601=5, srv656535658=80, srv236128661=50, srv527240190=74, srv702983858=83, srv966163484=94, srv390654907=65, srv1884848869=34, srv348615924=62, srv2021654246=42, srv1992810572=39, srv1192457820=8, srv127252726=11, srv919157665=91, srv1111616716=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:25,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 10:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 10:53:25,645 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0074477018519587814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009881452203147957); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0074477018519587814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:25,645 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 10:53:25,646 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.155776016550555E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0074477018519587814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009881452203147957); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0074477018519587814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 10:53:25,992 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 481 ms to try 5000 different iterations. Found a solution that moves 251 regions; Going from a computed imbalance of 8.155776016550555E-4 to a new imbalance of 4.993420421526068E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:25,993 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1143835153:83 , srv1709138241:83 , srv1958983033:83 , srv215070013:83 , srv397007354:83 , srv704525476:83 , srv745612046:83 , srv833742181:83 , srv1090407986:84 , srv1149975422:84 , srv1259441205:84 , srv1319757216:84 , srv1353945449:84 , srv1445802196:84 , srv1516273043:84 , srv1709244878:84 , srv1722504743:84 , srv2008299073:84 , srv2021654246:84 , srv2085478635:84 , srv2138354312:84 , srv264453119:84 , srv310914627:84 , srv327523118:84 , srv343381741:84 , srv367142132:84 , srv391227267:84 , srv50798747:84 , srv527240190:84 , srv581949915:84 , srv700745774:84 , srv761932927:84 , srv871668953:84 , srv919157665:84 , srv966163484:84 , srv1000583384:85 , srv106001744:85 , srv1109034982:85 , srv1111616716:85 , srv1119872601:85 , srv1192457820:85 , srv1202111045:85 , srv127252726:85 , srv1306672778:85 , srv1311828717:85 , srv1367980824:85 , srv1457326962:85 , srv1493646711:85 , srv1495277275:85 , srv151838048:85 , srv1596792077:85 , srv1680398819:85 , srv1768401386:85 , srv1779078570:85 , srv1815477317:85 , srv1834253353:85 , srv187795944:85 , srv1893576895:85 , srv1900187548:85 , srv2075022627:85 , srv2108477493:85 , srv214613420:85 , srv236128661:85 , srv239216161:85 , srv241454287:85 , srv299944746:85 , srv332028200:85 , srv345801564:85 , srv345831738:85 , srv348615924:85 , srv374041468:85 , srv390654907:85 , srv400706978:85 , srv40076264:85 , srv433850538:85 , srv454787009:85 , srv529267796:85 , srv610113787:85 , srv656535658:85 , srv700253227:85 , srv802162159:85 , srv817445393:85 , srv959781299:85 , srv964035624:85 , srv968099843:85 , srv968263924:85 , srv981034998:85 , srv988138455:85 , srv997271246:85 , srv182477840:86 , srv1884848869:86 , srv192133641:86 , srv1992810572:86 , srv2133899851:86 , srv266743310:86 , srv477272793:86 , srv551252176:86 , srv633228506:86 , srv702983858:86 , srv2017644481:618 } 2023-05-15 10:53:25,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:26,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv988138455=98, srv1893576895=35, srv1090407986=2, srv2075022627=43, srv2133899851=46, srv266743310=54, srv1779078570=29, srv581949915=77, srv332028200=58, srv1306672778=12, srv182477840=31, srv610113787=78, srv700745774=82, srv327523118=57, srv1493646711=19, srv50798747=73, srv529267796=75, srv968263924=96, srv700253227=81, srv997271246=99, srv310914627=56, srv214613420=48, srv1900187548=36, srv367142132=63, srv1367980824=16, srv400706978=68, srv1259441205=10, srv802162159=87, srv374041468=64, srv964035624=93, srv2138354312=47, srv477272793=72, srv215070013=49, srv397007354=67, srv1457326962=18, srv633228506=79, srv1596792077=23, srv345801564=60, srv299944746=55, srv2017644481=41, srv151838048=22, srv239216161=51, srv761932927=86, srv1680398819=24, srv551252176=76, srv968099843=95, srv2108477493=45, srv1311828717=13, srv192133641=37, srv704525476=84, srv959781299=92, srv343381741=59, srv1815477317=30, srv1000583384=0, srv433850538=70, srv1516273043=21, srv1709138241=25, srv40076264=69, srv981034998=97, srv1834253353=32, srv345831738=61, srv1495277275=20, srv1445802196=17, srv833742181=89, srv1709244878=26, srv454787009=71, srv1109034982=3, srv1149975422=7, srv187795944=33, srv871668953=90, srv241454287=52, srv1353945449=15, srv1722504743=27, srv1202111045=9, srv264453119=53, srv106001744=1, srv1768401386=28, srv2008299073=40, srv817445393=88, srv1958983033=38, srv2085478635=44, srv1143835153=6, srv1319757216=14, srv391227267=66, srv745612046=85, srv1119872601=5, srv656535658=80, srv236128661=50, srv527240190=74, srv702983858=83, srv966163484=94, srv390654907=65, srv1884848869=34, srv348615924=62, srv2021654246=42, srv1992810572=39, srv1192457820=8, srv127252726=11, srv919157665=91, srv1111616716=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:26,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:26,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 10:53:26,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 10:53:26,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 10:53:26,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 10:53:26,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 10:53:26,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 10:53:26,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 10:53:26,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 10:53:26,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 10:53:26,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 10:53:26,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 10:53:26,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 10:53:26,168 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:26,168 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 10:53:26,168 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.817666465061434E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 10:53:26,498 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 457 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 4.817666465061434E-5 to a new imbalance of 7.269299486807925E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:26,499 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1493646711:83 , srv1884848869:83 , srv1893576895:83 , srv236128661:83 , srv633228506:83 , srv702983858:83 , srv802162159:83 , srv964035624:83 , srv988138455:83 , srv997271246:83 , srv1000583384:84 , srv106001744:84 , srv1090407986:84 , srv1109034982:84 , srv1111616716:84 , srv1119872601:84 , srv1143835153:84 , srv1149975422:84 , srv1192457820:84 , srv1202111045:84 , srv1259441205:84 , srv127252726:84 , srv1306672778:84 , srv1311828717:84 , srv1319757216:84 , srv1353945449:84 , srv1367980824:84 , srv1445802196:84 , srv1495277275:84 , srv151838048:84 , srv1596792077:84 , srv1680398819:84 , srv1709244878:84 , srv1722504743:84 , srv1768401386:84 , srv1779078570:84 , srv1815477317:84 , srv182477840:84 , srv1834253353:84 , srv187795944:84 , srv1900187548:84 , srv192133641:84 , srv1992810572:84 , srv2021654246:84 , srv2075022627:84 , srv2085478635:84 , srv2108477493:84 , srv2133899851:84 , srv2138354312:84 , srv214613420:84 , srv239216161:84 , srv241454287:84 , srv264453119:84 , srv266743310:84 , srv299944746:84 , srv310914627:84 , srv332028200:84 , srv343381741:84 , srv345801564:84 , srv348615924:84 , srv367142132:84 , srv374041468:84 , srv390654907:84 , srv391227267:84 , srv397007354:84 , srv400706978:84 , srv40076264:84 , srv433850538:84 , srv454787009:84 , srv477272793:84 , srv527240190:84 , srv529267796:84 , srv551252176:84 , srv581949915:84 , srv610113787:84 , srv656535658:84 , srv700253227:84 , srv700745774:84 , srv704525476:84 , srv745612046:84 , srv761932927:84 , srv817445393:84 , srv871668953:84 , srv919157665:84 , srv959781299:84 , srv966163484:84 , srv968099843:84 , srv968263924:84 , srv981034998:84 , srv1457326962:85 , srv1516273043:85 , srv1709138241:85 , srv1958983033:85 , srv2008299073:85 , srv215070013:85 , srv327523118:85 , srv345831738:85 , srv50798747:85 , srv833742181:85 , srv2017644481:684 } 2023-05-15 10:53:26,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:26,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv988138455=98, srv1893576895=35, srv1090407986=2, srv2075022627=43, srv2133899851=46, srv266743310=54, srv1779078570=29, srv581949915=77, srv332028200=58, srv1306672778=12, srv182477840=31, srv610113787=78, srv700745774=82, srv327523118=57, srv1493646711=19, srv50798747=73, srv529267796=75, srv968263924=96, srv700253227=81, srv997271246=99, srv310914627=56, srv214613420=48, srv1900187548=36, srv367142132=63, srv1367980824=16, srv400706978=68, srv1259441205=10, srv802162159=87, srv374041468=64, srv964035624=93, srv2138354312=47, srv477272793=72, srv215070013=49, srv397007354=67, srv1457326962=18, srv633228506=79, srv1596792077=23, srv345801564=60, srv299944746=55, srv2017644481=41, srv151838048=22, srv239216161=51, srv761932927=86, srv1680398819=24, srv551252176=76, srv968099843=95, srv2108477493=45, srv1311828717=13, srv192133641=37, srv704525476=84, srv959781299=92, srv343381741=59, srv1815477317=30, srv1000583384=0, srv433850538=70, srv1516273043=21, srv1709138241=25, srv40076264=69, srv981034998=97, srv1834253353=32, srv345831738=61, srv1495277275=20, srv1445802196=17, srv833742181=89, srv1709244878=26, srv454787009=71, srv1109034982=3, srv1149975422=7, srv187795944=33, srv871668953=90, srv241454287=52, srv1353945449=15, srv1722504743=27, srv1202111045=9, srv264453119=53, srv106001744=1, srv1768401386=28, srv2008299073=40, srv817445393=88, srv1958983033=38, srv2085478635=44, srv1143835153=6, srv1319757216=14, srv391227267=66, srv745612046=85, srv1119872601=5, srv656535658=80, srv236128661=50, srv527240190=74, srv702983858=83, srv966163484=94, srv390654907=65, srv1884848869=34, srv348615924=62, srv2021654246=42, srv1992810572=39, srv1192457820=8, srv127252726=11, srv919157665=91, srv1111616716=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:26,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:26,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:26,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:26,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:26,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 10:53:26,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 10:53:26,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 10:53:26,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 10:53:26,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 10:53:26,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 10:53:26,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 10:53:26,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 10:53:26,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:26,653 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1493646711:83 , srv1884848869:83 , srv1893576895:83 , srv236128661:83 , srv633228506:83 , srv702983858:83 , srv802162159:83 , srv964035624:83 , srv988138455:83 , srv997271246:83 , srv1000583384:84 , srv106001744:84 , srv1090407986:84 , srv1109034982:84 , srv1111616716:84 , srv1119872601:84 , srv1143835153:84 , srv1149975422:84 , srv1192457820:84 , srv1202111045:84 , srv1259441205:84 , srv127252726:84 , srv1306672778:84 , srv1311828717:84 , srv1319757216:84 , srv1353945449:84 , srv1367980824:84 , srv1445802196:84 , srv1495277275:84 , srv151838048:84 , srv1596792077:84 , srv1680398819:84 , srv1709244878:84 , srv1722504743:84 , srv1768401386:84 , srv1779078570:84 , srv1815477317:84 , srv182477840:84 , srv1834253353:84 , srv187795944:84 , srv1900187548:84 , srv192133641:84 , srv1992810572:84 , srv2021654246:84 , srv2075022627:84 , srv2085478635:84 , srv2108477493:84 , srv2133899851:84 , srv2138354312:84 , srv214613420:84 , srv239216161:84 , srv241454287:84 , srv264453119:84 , srv266743310:84 , srv299944746:84 , srv310914627:84 , srv332028200:84 , srv343381741:84 , srv345801564:84 , srv348615924:84 , srv367142132:84 , srv374041468:84 , srv390654907:84 , srv391227267:84 , srv397007354:84 , srv400706978:84 , srv40076264:84 , srv433850538:84 , srv454787009:84 , srv477272793:84 , srv527240190:84 , srv529267796:84 , srv551252176:84 , srv581949915:84 , srv610113787:84 , srv656535658:84 , srv700253227:84 , srv700745774:84 , srv704525476:84 , srv745612046:84 , srv761932927:84 , srv817445393:84 , srv871668953:84 , srv919157665:84 , srv959781299:84 , srv966163484:84 , srv968099843:84 , srv968263924:84 , srv981034998:84 , srv1457326962:85 , srv1516273043:85 , srv1709138241:85 , srv1958983033:85 , srv2008299073:85 , srv215070013:85 , srv327523118:85 , srv345831738:85 , srv50798747:85 , srv833742181:85 , srv2017644481:684 } 2023-05-15 10:53:26,688 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=326 (was 326), ProcessCount=171 (was 171), AvailableMemoryMB=14173 (was 14905) 2023-05-15 10:53:26,702 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=171, AvailableMemoryMB=14171 2023-05-15 10:53:26,703 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 10:53:26,703 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-15 10:53:26,714 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1053665711:84 , srv1108200138:84 , srv1284070637:84 , srv1301980942:84 , srv1328841457:84 , srv1339690805:84 , srv1400725738:84 , srv1406927883:84 , srv1478505518:84 , srv1552982352:84 , srv1567930687:84 , srv1613211098:84 , srv1632970785:84 , srv1797928208:84 , srv1837659662:84 , srv1921857332:84 , srv1947825863:84 , srv1956713847:84 , srv1959828604:84 , srv2037292980:84 , srv2144783572:84 , srv49833239:84 , srv640633246:84 , srv660572139:84 , srv72916886:84 , srv851229732:84 , srv897190304:84 , srv947750568:84 , srv958319403:84 , srv28032406:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-15 10:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1567930687=10, srv947750568=28, srv1956713847=17, srv1959828604=18, srv897190304=27, srv1613211098=11, srv1632970785=12, srv2144783572=20, srv660572139=24, srv958319403=29, srv1797928208=13, srv1301980942=3, srv1328841457=4, srv49833239=22, srv72916886=25, srv1053665711=0, srv1400725738=6, srv1552982352=9, srv1921857332=15, srv640633246=23, srv1339690805=5, srv1284070637=2, srv1478505518=8, srv28032406=21, srv1406927883=7, srv851229732=26, srv1108200138=1, srv2037292980=19, srv1947825863=16, srv1837659662=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:53:26,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-15 10:53:26,743 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1191102891548115, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:53:26,744 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912568355896098, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1191102891548115, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-15 10:55:26,730 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6184753 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912568355896098 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:55:26,737 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1053665711:90 , srv1108200138:90 , srv1284070637:90 , srv1301980942:90 , srv1328841457:90 , srv1339690805:90 , srv1400725738:90 , srv1406927883:90 , srv1478505518:90 , srv1552982352:90 , srv1567930687:90 , srv1613211098:90 , srv1632970785:90 , srv1797928208:90 , srv1837659662:90 , srv1921857332:90 , srv1947825863:90 , srv1956713847:90 , srv1959828604:90 , srv2037292980:90 , srv2144783572:90 , srv28032406:90 , srv49833239:90 , srv640633246:90 , srv660572139:90 , srv72916886:90 , srv851229732:90 , srv897190304:90 , srv947750568:90 , srv958319403:90 } 2023-05-15 10:55:26,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 10:55:26,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1567930687=10, srv947750568=28, srv1956713847=17, srv1959828604=18, srv897190304=27, srv1613211098=11, srv1632970785=12, srv2144783572=20, srv660572139=24, srv958319403=29, srv1797928208=13, srv1301980942=3, srv1328841457=4, srv49833239=22, srv72916886=25, srv1053665711=0, srv1400725738=6, srv1552982352=9, srv1921857332=15, srv640633246=23, srv1339690805=5, srv1284070637=2, srv1478505518=8, srv28032406=21, srv1406927883=7, srv851229732=26, srv1108200138=1, srv2037292980=19, srv1947825863=16, srv1837659662=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 10:55:26,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 10:55:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 10:55:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-15 10:55:26,762 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 10:55:26,763 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1053665711:90 , srv1108200138:90 , srv1284070637:90 , srv1301980942:90 , srv1328841457:90 , srv1339690805:90 , srv1400725738:90 , srv1406927883:90 , srv1478505518:90 , srv1552982352:90 , srv1567930687:90 , srv1613211098:90 , srv1632970785:90 , srv1797928208:90 , srv1837659662:90 , srv1921857332:90 , srv1947825863:90 , srv1956713847:90 , srv1959828604:90 , srv2037292980:90 , srv2144783572:90 , srv28032406:90 , srv49833239:90 , srv640633246:90 , srv660572139:90 , srv72916886:90 , srv851229732:90 , srv897190304:90 , srv947750568:90 , srv958319403:90 } 2023-05-15 10:55:26,780 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=141 (was 326), ProcessCount=169 (was 171), AvailableMemoryMB=14142 (was 14171)