2023-10-11 16:53:26,269 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-11 16:53:26,548 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-11 16:53:26,647 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-11 16:53:26,709 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-11 16:53:26,716 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-11 16:53:26,729 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=171, AvailableMemoryMB=6637 2023-10-11 16:53:26,731 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-11 16:53:26,731 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-11 16:53:26,867 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016087569:84 , srv1020808623:84 , srv1021655348:84 , srv1031476304:84 , srv1040364476:84 , srv104331589:84 , srv1052351169:84 , srv105269263:84 , srv1058512945:84 , srv1080731608:84 , srv1125585439:84 , srv1134008437:84 , srv1154416672:84 , srv1167114386:84 , srv1171188585:84 , srv118764955:84 , srv1214494252:84 , srv1221210998:84 , srv1226926686:84 , srv1244873975:84 , srv1257877016:84 , srv12666098:84 , srv1278220565:84 , srv1289793660:84 , srv1297866854:84 , srv1308656614:84 , srv1338151219:84 , srv1365571503:84 , srv1391596147:84 , srv142395237:84 , srv1506717936:84 , srv1560296651:84 , srv1568768539:84 , srv1618145457:84 , srv1635166200:84 , srv164906284:84 , srv1669268617:84 , srv1704524347:84 , srv1710294692:84 , srv1724264016:84 , srv174360105:84 , srv1746208879:84 , srv1775868100:84 , srv1861625149:84 , srv1876892606:84 , srv1901597236:84 , srv1908990207:84 , srv1921946002:84 , srv1925424017:84 , srv1937662969:84 , srv1954480221:84 , srv2015873695:84 , srv2031061527:84 , srv2053851569:84 , srv208511088:84 , srv2094503690:84 , srv215809107:84 , srv241385811:84 , srv250599473:84 , srv266237744:84 , srv283423249:84 , srv283734874:84 , srv317271729:84 , srv31909970:84 , srv320487659:84 , srv345335834:84 , srv372256797:84 , srv37593444:84 , srv38084764:84 , srv395819388:84 , srv416564834:84 , srv432673553:84 , srv465140754:84 , srv466823732:84 , srv474948006:84 , srv486153366:84 , srv488988563:84 , srv521501575:84 , srv524095396:84 , srv576516028:84 , srv57669160:84 , srv58297225:84 , srv613899165:84 , srv614938477:84 , srv619157000:84 , srv652539146:84 , srv654986258:84 , srv680324397:84 , srv690688190:84 , srv708549152:84 , srv739731420:84 , srv758330661:84 , srv763323298:84 , srv787214682:84 , srv838756306:84 , srv882758420:84 , srv942337035:84 , srv958138367:84 , srv983810911:84 , srv1718468310:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-11 16:53:26,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:26,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv395819388=70, srv614938477=84, srv250599473=59, srv465140754=73, srv486153366=76, srv1221210998=17, srv983810911=99, srv576516028=80, srv1125585439=10, srv320487659=65, srv474948006=75, srv416564834=71, srv690688190=89, srv942337035=97, srv241385811=58, srv104331589=5, srv882758420=96, srv1257877016=20, srv1058512945=8, srv524095396=79, srv174360105=41, srv12666098=21, srv1154416672=12, srv758330661=92, srv488988563=77, srv1724264016=40, srv466823732=74, srv1718468310=39, srv1021655348=2, srv1861625149=44, srv1308656614=25, srv1040364476=4, srv317271729=63, srv37593444=68, srv1080731608=9, srv1925424017=49, srv1901597236=46, srv1391596147=28, srv1365571503=27, srv2053851569=54, srv1560296651=31, srv38084764=69, srv1020808623=1, srv1214494252=16, srv1568768539=32, srv164906284=35, srv432673553=72, srv1226926686=18, srv2094503690=56, srv1278220565=22, srv1134008437=11, srv31909970=64, srv1635166200=34, srv1506717936=30, srv1618145457=33, srv739731420=91, srv787214682=94, srv1052351169=6, srv958138367=98, srv1775868100=43, srv838756306=95, srv1704524347=37, srv613899165=83, srv283734874=62, srv142395237=29, srv1289793660=23, srv208511088=55, srv2031061527=53, srv1031476304=3, srv215809107=57, srv118764955=15, srv654986258=87, srv1338151219=26, srv1921946002=48, srv708549152=90, srv1669268617=36, srv1908990207=47, srv763323298=93, srv2015873695=52, srv372256797=67, srv521501575=78, srv105269263=7, srv1171188585=14, srv266237744=60, srv1297866854=24, srv1167114386=13, srv57669160=81, srv1016087569=0, srv1244873975=19, srv283423249=61, srv680324397=88, srv1710294692=38, srv619157000=85, srv652539146=86, srv1876892606=45, srv1954480221=51, srv345335834=66, srv58297225=82, srv1937662969=50, srv1746208879=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:26,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-11 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-11 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-11 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-11 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-11 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-11 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-11 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-11 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-11 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-11 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-11 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-11 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-11 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-11 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-11 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-11 16:53:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-11 16:53:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-11 16:53:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-11 16:53:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-11 16:53:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-11 16:53:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-11 16:53:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-11 16:53:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-11 16:53:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-11 16:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-11 16:53:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-11 16:53:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-11 16:53:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-11 16:53:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-11 16:53:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-11 16:53:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-11 16:53:26,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-11 16:53:27,263 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:27,263 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-11 16:53:27,263 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-11 16:53:27,627 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 624 ms to try 5000 different iterations. Found a solution that moves 4694 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17726291230698388. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04049635917736029, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055698805214806024, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5215555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04049635917736029, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:27,652 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1669268617:82 , srv320487659:82 , srv345335834:82 , srv942337035:82 , srv1031476304:83 , srv1134008437:83 , srv1154416672:83 , srv1214494252:83 , srv1937662969:83 , srv38084764:83 , srv488988563:83 , srv838756306:83 , srv1040364476:84 , srv1365571503:84 , srv1391596147:84 , srv1746208879:84 , srv1861625149:84 , srv31909970:84 , srv432673553:84 , srv787214682:84 , srv1020808623:85 , srv1167114386:85 , srv1226926686:85 , srv1338151219:85 , srv1506717936:85 , srv1724264016:85 , srv1901597236:85 , srv2053851569:85 , srv215809107:85 , srv250599473:85 , srv317271729:85 , srv37593444:85 , srv416564834:85 , srv619157000:85 , srv652539146:85 , srv680324397:85 , srv708549152:85 , srv958138367:85 , srv105269263:86 , srv1058512945:86 , srv1125585439:86 , srv1244873975:86 , srv12666098:86 , srv1289793660:86 , srv1297866854:86 , srv142395237:86 , srv1704524347:86 , srv1710294692:86 , srv1908990207:86 , srv2015873695:86 , srv2094503690:86 , srv241385811:86 , srv283734874:86 , srv465140754:86 , srv466823732:86 , srv474948006:86 , srv614938477:86 , srv739731420:86 , srv758330661:86 , srv1171188585:87 , srv1221210998:87 , srv1257877016:87 , srv1560296651:87 , srv1775868100:87 , srv1925424017:87 , srv1954480221:87 , srv2031061527:87 , srv208511088:87 , srv372256797:87 , srv524095396:87 , srv576516028:87 , srv58297225:87 , srv882758420:87 , srv1052351169:88 , srv1876892606:88 , srv395819388:88 , srv690688190:88 , srv763323298:88 , srv1016087569:89 , srv1021655348:89 , srv118764955:89 , srv1278220565:89 , srv1308656614:89 , srv1568768539:89 , srv283423249:89 , srv486153366:89 , srv521501575:89 , srv104331589:90 , srv1080731608:90 , srv1618145457:90 , srv1635166200:90 , srv57669160:90 , srv613899165:90 , srv654986258:90 , srv174360105:91 , srv983810911:91 , srv266237744:92 , srv164906284:93 , srv1921946002:94 , srv1718468310:450 } 2023-10-11 16:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv395819388=70, srv614938477=84, srv250599473=59, srv465140754=73, srv486153366=76, srv1221210998=17, srv983810911=99, srv576516028=80, srv1125585439=10, srv320487659=65, srv474948006=75, srv416564834=71, srv690688190=89, srv942337035=97, srv241385811=58, srv104331589=5, srv882758420=96, srv1257877016=20, srv1058512945=8, srv524095396=79, srv174360105=41, srv12666098=21, srv1154416672=12, srv758330661=92, srv488988563=77, srv1724264016=40, srv466823732=74, srv1718468310=39, srv1021655348=2, srv1861625149=44, srv1308656614=25, srv1040364476=4, srv317271729=63, srv37593444=68, srv1080731608=9, srv1925424017=49, srv1901597236=46, srv1391596147=28, srv1365571503=27, srv2053851569=54, srv1560296651=31, srv38084764=69, srv1020808623=1, srv1214494252=16, srv1568768539=32, srv164906284=35, srv432673553=72, srv1226926686=18, srv2094503690=56, srv1278220565=22, srv1134008437=11, srv31909970=64, srv1635166200=34, srv1506717936=30, srv1618145457=33, srv739731420=91, srv787214682=94, srv1052351169=6, srv958138367=98, srv1775868100=43, srv838756306=95, srv1704524347=37, srv613899165=83, srv283734874=62, srv142395237=29, srv1289793660=23, srv208511088=55, srv2031061527=53, srv1031476304=3, srv215809107=57, srv118764955=15, srv654986258=87, srv1338151219=26, srv1921946002=48, srv708549152=90, srv1669268617=36, srv1908990207=47, srv763323298=93, srv2015873695=52, srv372256797=67, srv521501575=78, srv105269263=7, srv1171188585=14, srv266237744=60, srv1297866854=24, srv1167114386=13, srv57669160=81, srv1016087569=0, srv1244873975=19, srv283423249=61, srv680324397=88, srv1710294692=38, srv619157000=85, srv652539146=86, srv1876892606=45, srv1954480221=51, srv345335834=66, srv58297225=82, srv1937662969=50, srv1746208879=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:27,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:27,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:27,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:27,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:27,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:27,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:27,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:27,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:27,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:27,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:27,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:27,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:27,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:27,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:27,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:27,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:27,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:27,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:27,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:27,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:27,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:27,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:27,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:27,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:27,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:27,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:27,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-11 16:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-11 16:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-11 16:53:27,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-11 16:53:27,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-11 16:53:27,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-11 16:53:27,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-11 16:53:27,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-11 16:53:27,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-11 16:53:27,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-11 16:53:27,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-11 16:53:27,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-11 16:53:27,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-11 16:53:27,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-11 16:53:27,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-11 16:53:27,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-11 16:53:27,945 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04049635917736029, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055698805214806024, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04049635917736029, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:27,945 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-11 16:53:27,945 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1772300442164801, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04049635917736029, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055698805214806024, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04049635917736029, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-11 16:53:28,418 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 675 ms to try 5000 different iterations. Found a solution that moves 2696 regions; Going from a computed imbalance of 0.1772300442164801 to a new imbalance of 0.015242846266662032. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020606225654584264); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03001683029585404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2995555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020606225654584264); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:28,431 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1921946002:80 , srv266237744:80 , srv1052351169:81 , srv1080731608:81 , srv164906284:81 , srv57669160:81 , srv983810911:81 , srv118764955:82 , srv1221210998:82 , srv174360105:82 , srv395819388:82 , srv521501575:82 , srv104331589:83 , srv1257877016:83 , srv1278220565:83 , srv1560296651:83 , srv1618145457:83 , srv1635166200:83 , srv1775868100:83 , srv690688190:83 , srv1171188585:84 , srv1308656614:84 , srv1568768539:84 , srv283423249:84 , srv486153366:84 , srv524095396:84 , srv576516028:84 , srv613899165:84 , srv654986258:84 , srv1058512945:85 , srv1125585439:85 , srv1244873975:85 , srv1704524347:85 , srv1908990207:85 , srv1925424017:85 , srv1954480221:85 , srv2015873695:85 , srv208511088:85 , srv241385811:85 , srv283734874:85 , srv416564834:85 , srv465140754:85 , srv739731420:85 , srv763323298:85 , srv1020808623:86 , srv1021655348:86 , srv1040364476:86 , srv105269263:86 , srv12666098:86 , srv1289793660:86 , srv1861625149:86 , srv1876892606:86 , srv2031061527:86 , srv2094503690:86 , srv466823732:86 , srv708549152:86 , srv882758420:86 , srv958138367:86 , srv1016087569:87 , srv1226926686:87 , srv1297866854:87 , srv142395237:87 , srv1901597236:87 , srv1937662969:87 , srv317271729:87 , srv345335834:87 , srv372256797:87 , srv37593444:87 , srv474948006:87 , srv58297225:87 , srv614938477:87 , srv680324397:87 , srv1134008437:88 , srv1338151219:88 , srv1365571503:88 , srv1506717936:88 , srv1710294692:88 , srv1724264016:88 , srv2053851569:88 , srv215809107:88 , srv320487659:88 , srv432673553:88 , srv488988563:88 , srv619157000:88 , srv758330661:88 , srv838756306:88 , srv1031476304:89 , srv1391596147:89 , srv1669268617:89 , srv250599473:89 , srv31909970:89 , srv1214494252:90 , srv1746208879:90 , srv787214682:90 , srv942337035:90 , srv1154416672:91 , srv1167114386:91 , srv38084764:91 , srv652539146:91 , srv1718468310:507 } 2023-10-11 16:53:28,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:28,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv395819388=70, srv614938477=84, srv250599473=59, srv465140754=73, srv486153366=76, srv1221210998=17, srv983810911=99, srv576516028=80, srv1125585439=10, srv320487659=65, srv474948006=75, srv416564834=71, srv690688190=89, srv942337035=97, srv241385811=58, srv104331589=5, srv882758420=96, srv1257877016=20, srv1058512945=8, srv524095396=79, srv174360105=41, srv12666098=21, srv1154416672=12, srv758330661=92, srv488988563=77, srv1724264016=40, srv466823732=74, srv1718468310=39, srv1021655348=2, srv1861625149=44, srv1308656614=25, srv1040364476=4, srv317271729=63, srv37593444=68, srv1080731608=9, srv1925424017=49, srv1901597236=46, srv1391596147=28, srv1365571503=27, srv2053851569=54, srv1560296651=31, srv38084764=69, srv1020808623=1, srv1214494252=16, srv1568768539=32, srv164906284=35, srv432673553=72, srv1226926686=18, srv2094503690=56, srv1278220565=22, srv1134008437=11, srv31909970=64, srv1635166200=34, srv1506717936=30, srv1618145457=33, srv739731420=91, srv787214682=94, srv1052351169=6, srv958138367=98, srv1775868100=43, srv838756306=95, srv1704524347=37, srv613899165=83, srv283734874=62, srv142395237=29, srv1289793660=23, srv208511088=55, srv2031061527=53, srv1031476304=3, srv215809107=57, srv118764955=15, srv654986258=87, srv1338151219=26, srv1921946002=48, srv708549152=90, srv1669268617=36, srv1908990207=47, srv763323298=93, srv2015873695=52, srv372256797=67, srv521501575=78, srv105269263=7, srv1171188585=14, srv266237744=60, srv1297866854=24, srv1167114386=13, srv57669160=81, srv1016087569=0, srv1244873975=19, srv283423249=61, srv680324397=88, srv1710294692=38, srv619157000=85, srv652539146=86, srv1876892606=45, srv1954480221=51, srv345335834=66, srv58297225=82, srv1937662969=50, srv1746208879=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:28,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:28,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:28,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:28,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:28,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:28,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-11 16:53:28,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-11 16:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-11 16:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-11 16:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-11 16:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-11 16:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-11 16:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-11 16:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-11 16:53:28,635 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020606225654584264); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03001683029585404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020606225654584264); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:28,635 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-11 16:53:28,635 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015223968471178819, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020606225654584264); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03001683029585404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020606225654584264); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-11 16:53:28,987 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 489 ms to try 5000 different iterations. Found a solution that moves 957 regions; Going from a computed imbalance of 0.015223968471178819 to a new imbalance of 9.647488996633312E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005206785216983458); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.10633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005206785216983458); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:28,991 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1016087569:81 , srv1167114386:82 , srv652539146:82 , srv1921946002:83 , srv372256797:83 , srv613899165:83 , srv654986258:83 , srv1021655348:84 , srv1308656614:84 , srv1338151219:84 , srv142395237:84 , srv1506717936:84 , srv1635166200:84 , srv164906284:84 , srv1710294692:84 , srv1746208879:84 , srv1876892606:84 , srv2031061527:84 , srv215809107:84 , srv250599473:84 , srv283423249:84 , srv38084764:84 , srv474948006:84 , srv58297225:84 , srv758330661:84 , srv763323298:84 , srv787214682:84 , srv104331589:85 , srv105269263:85 , srv1154416672:85 , srv1214494252:85 , srv1226926686:85 , srv1289793660:85 , srv1297866854:85 , srv1365571503:85 , srv1391596147:85 , srv1568768539:85 , srv1618145457:85 , srv1724264016:85 , srv174360105:85 , srv2015873695:85 , srv2053851569:85 , srv208511088:85 , srv2094503690:85 , srv317271729:85 , srv31909970:85 , srv37593444:85 , srv465140754:85 , srv466823732:85 , srv486153366:85 , srv614938477:85 , srv619157000:85 , srv680324397:85 , srv882758420:85 , srv942337035:85 , srv983810911:85 , srv1031476304:86 , srv1058512945:86 , srv1125585439:86 , srv12666098:86 , srv1278220565:86 , srv1669268617:86 , srv1704524347:86 , srv1908990207:86 , srv1925424017:86 , srv1954480221:86 , srv266237744:86 , srv283734874:86 , srv432673553:86 , srv521501575:86 , srv524095396:86 , srv576516028:86 , srv57669160:86 , srv739731420:86 , srv958138367:86 , srv1020808623:87 , srv1040364476:87 , srv1134008437:87 , srv1171188585:87 , srv118764955:87 , srv1244873975:87 , srv1257877016:87 , srv1560296651:87 , srv1775868100:87 , srv1901597236:87 , srv1937662969:87 , srv241385811:87 , srv320487659:87 , srv488988563:87 , srv690688190:87 , srv708549152:87 , srv838756306:87 , srv1052351169:88 , srv1080731608:88 , srv1861625149:88 , srv395819388:88 , srv416564834:88 , srv1221210998:89 , srv345335834:89 , srv1718468310:547 } 2023-10-11 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv395819388=70, srv614938477=84, srv250599473=59, srv465140754=73, srv486153366=76, srv1221210998=17, srv983810911=99, srv576516028=80, srv1125585439=10, srv320487659=65, srv474948006=75, srv416564834=71, srv690688190=89, srv942337035=97, srv241385811=58, srv104331589=5, srv882758420=96, srv1257877016=20, srv1058512945=8, srv524095396=79, srv174360105=41, srv12666098=21, srv1154416672=12, srv758330661=92, srv488988563=77, srv1724264016=40, srv466823732=74, srv1718468310=39, srv1021655348=2, srv1861625149=44, srv1308656614=25, srv1040364476=4, srv317271729=63, srv37593444=68, srv1080731608=9, srv1925424017=49, srv1901597236=46, srv1391596147=28, srv1365571503=27, srv2053851569=54, srv1560296651=31, srv38084764=69, srv1020808623=1, srv1214494252=16, srv1568768539=32, srv164906284=35, srv432673553=72, srv1226926686=18, srv2094503690=56, srv1278220565=22, srv1134008437=11, srv31909970=64, srv1635166200=34, srv1506717936=30, srv1618145457=33, srv739731420=91, srv787214682=94, srv1052351169=6, srv958138367=98, srv1775868100=43, srv838756306=95, srv1704524347=37, srv613899165=83, srv283734874=62, srv142395237=29, srv1289793660=23, srv208511088=55, srv2031061527=53, srv1031476304=3, srv215809107=57, srv118764955=15, srv654986258=87, srv1338151219=26, srv1921946002=48, srv708549152=90, srv1669268617=36, srv1908990207=47, srv763323298=93, srv2015873695=52, srv372256797=67, srv521501575=78, srv105269263=7, srv1171188585=14, srv266237744=60, srv1297866854=24, srv1167114386=13, srv57669160=81, srv1016087569=0, srv1244873975=19, srv283423249=61, srv680324397=88, srv1710294692=38, srv619157000=85, srv652539146=86, srv1876892606=45, srv1954480221=51, srv345335834=66, srv58297225=82, srv1937662969=50, srv1746208879=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:29,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:29,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:29,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-11 16:53:29,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-11 16:53:29,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-11 16:53:29,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-11 16:53:29,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-11 16:53:29,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-11 16:53:29,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-11 16:53:29,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-11 16:53:29,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-11 16:53:29,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-11 16:53:29,142 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005206785216983458); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005206785216983458); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:29,142 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-11 16:53:29,143 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.580478424387632E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005206785216983458); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005206785216983458); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-11 16:53:29,593 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 560 ms to try 5000 different iterations. Found a solution that moves 223 regions; Going from a computed imbalance of 9.580478424387632E-4 to a new imbalance of 2.75107174448256E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.024777777777777777); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:29,594 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1021655348:83 , srv1080731608:83 , srv1154416672:83 , srv164906284:83 , srv1901597236:83 , srv2053851569:83 , srv1020808623:84 , srv1031476304:84 , srv104331589:84 , srv105269263:84 , srv1134008437:84 , srv1171188585:84 , srv118764955:84 , srv1244873975:84 , srv12666098:84 , srv1278220565:84 , srv1289793660:84 , srv1297866854:84 , srv1391596147:84 , srv1618145457:84 , srv1710294692:84 , srv1724264016:84 , srv174360105:84 , srv1746208879:84 , srv1861625149:84 , srv1876892606:84 , srv1925424017:84 , srv1954480221:84 , srv208511088:84 , srv241385811:84 , srv250599473:84 , srv266237744:84 , srv317271729:84 , srv31909970:84 , srv345335834:84 , srv38084764:84 , srv395819388:84 , srv416564834:84 , srv432673553:84 , srv486153366:84 , srv488988563:84 , srv58297225:84 , srv614938477:84 , srv619157000:84 , srv652539146:84 , srv654986258:84 , srv690688190:84 , srv708549152:84 , srv758330661:84 , srv787214682:84 , srv838756306:84 , srv983810911:84 , srv1016087569:85 , srv1040364476:85 , srv1052351169:85 , srv1058512945:85 , srv1125585439:85 , srv1167114386:85 , srv1214494252:85 , srv1221210998:85 , srv1226926686:85 , srv1257877016:85 , srv1308656614:85 , srv1338151219:85 , srv1365571503:85 , srv142395237:85 , srv1506717936:85 , srv1560296651:85 , srv1568768539:85 , srv1635166200:85 , srv1669268617:85 , srv1704524347:85 , srv1775868100:85 , srv1908990207:85 , srv1921946002:85 , srv1937662969:85 , srv2031061527:85 , srv2094503690:85 , srv215809107:85 , srv283423249:85 , srv283734874:85 , srv320487659:85 , srv372256797:85 , srv37593444:85 , srv474948006:85 , srv521501575:85 , srv524095396:85 , srv57669160:85 , srv613899165:85 , srv680324397:85 , srv739731420:85 , srv763323298:85 , srv882758420:85 , srv942337035:85 , srv2015873695:86 , srv465140754:86 , srv466823732:86 , srv576516028:86 , srv958138367:86 , srv1718468310:638 } 2023-10-11 16:53:29,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:29,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv395819388=70, srv614938477=84, srv250599473=59, srv465140754=73, srv486153366=76, srv1221210998=17, srv983810911=99, srv576516028=80, srv1125585439=10, srv320487659=65, srv474948006=75, srv416564834=71, srv690688190=89, srv942337035=97, srv241385811=58, srv104331589=5, srv882758420=96, srv1257877016=20, srv1058512945=8, srv524095396=79, srv174360105=41, srv12666098=21, srv1154416672=12, srv758330661=92, srv488988563=77, srv1724264016=40, srv466823732=74, srv1718468310=39, srv1021655348=2, srv1861625149=44, srv1308656614=25, srv1040364476=4, srv317271729=63, srv37593444=68, srv1080731608=9, srv1925424017=49, srv1901597236=46, srv1391596147=28, srv1365571503=27, srv2053851569=54, srv1560296651=31, srv38084764=69, srv1020808623=1, srv1214494252=16, srv1568768539=32, srv164906284=35, srv432673553=72, srv1226926686=18, srv2094503690=56, srv1278220565=22, srv1134008437=11, srv31909970=64, srv1635166200=34, srv1506717936=30, srv1618145457=33, srv739731420=91, srv787214682=94, srv1052351169=6, srv958138367=98, srv1775868100=43, srv838756306=95, srv1704524347=37, srv613899165=83, srv283734874=62, srv142395237=29, srv1289793660=23, srv208511088=55, srv2031061527=53, srv1031476304=3, srv215809107=57, srv118764955=15, srv654986258=87, srv1338151219=26, srv1921946002=48, srv708549152=90, srv1669268617=36, srv1908990207=47, srv763323298=93, srv2015873695=52, srv372256797=67, srv521501575=78, srv105269263=7, srv1171188585=14, srv266237744=60, srv1297866854=24, srv1167114386=13, srv57669160=81, srv1016087569=0, srv1244873975=19, srv283423249=61, srv680324397=88, srv1710294692=38, srv619157000=85, srv652539146=86, srv1876892606=45, srv1954480221=51, srv345335834=66, srv58297225=82, srv1937662969=50, srv1746208879=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-11 16:53:29,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-11 16:53:29,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-11 16:53:29,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-11 16:53:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-11 16:53:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-11 16:53:29,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-11 16:53:29,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-11 16:53:29,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-11 16:53:29,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-11 16:53:29,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-11 16:53:29,786 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:29,786 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-11 16:53:29,786 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.5949238070657733E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-11 16:53:30,092 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 454 ms to try 5000 different iterations. Found a solution that moves 22 regions; Going from a computed imbalance of 2.5949238070657733E-5 to a new imbalance of 7.541804792831362E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0024444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:30,092 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1167114386:83 , srv1214494252:83 , srv1221210998:83 , srv1568768539:83 , srv466823732:83 , srv576516028:83 , srv882758420:83 , srv958138367:83 , srv1016087569:84 , srv1020808623:84 , srv1021655348:84 , srv1031476304:84 , srv1040364476:84 , srv104331589:84 , srv1052351169:84 , srv1058512945:84 , srv1080731608:84 , srv1125585439:84 , srv1134008437:84 , srv1154416672:84 , srv1171188585:84 , srv118764955:84 , srv1226926686:84 , srv1244873975:84 , srv1257877016:84 , srv12666098:84 , srv1278220565:84 , srv1297866854:84 , srv1308656614:84 , srv1338151219:84 , srv1365571503:84 , srv1391596147:84 , srv142395237:84 , srv1506717936:84 , srv1560296651:84 , srv1618145457:84 , srv1635166200:84 , srv1669268617:84 , srv1704524347:84 , srv1710294692:84 , srv1724264016:84 , srv174360105:84 , srv1746208879:84 , srv1775868100:84 , srv1861625149:84 , srv1876892606:84 , srv1901597236:84 , srv1908990207:84 , srv1921946002:84 , srv1925424017:84 , srv1937662969:84 , srv1954480221:84 , srv2015873695:84 , srv2031061527:84 , srv2094503690:84 , srv215809107:84 , srv241385811:84 , srv250599473:84 , srv266237744:84 , srv283423249:84 , srv283734874:84 , srv31909970:84 , srv320487659:84 , srv345335834:84 , srv372256797:84 , srv37593444:84 , srv38084764:84 , srv395819388:84 , srv416564834:84 , srv432673553:84 , srv465140754:84 , srv474948006:84 , srv486153366:84 , srv488988563:84 , srv521501575:84 , srv524095396:84 , srv57669160:84 , srv58297225:84 , srv613899165:84 , srv614938477:84 , srv619157000:84 , srv652539146:84 , srv680324397:84 , srv690688190:84 , srv708549152:84 , srv739731420:84 , srv758330661:84 , srv763323298:84 , srv787214682:84 , srv838756306:84 , srv942337035:84 , srv105269263:85 , srv1289793660:85 , srv164906284:85 , srv2053851569:85 , srv208511088:85 , srv317271729:85 , srv654986258:85 , srv983810911:85 , srv1718468310:684 } 2023-10-11 16:53:30,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:30,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv395819388=70, srv614938477=84, srv250599473=59, srv465140754=73, srv486153366=76, srv1221210998=17, srv983810911=99, srv576516028=80, srv1125585439=10, srv320487659=65, srv474948006=75, srv416564834=71, srv690688190=89, srv942337035=97, srv241385811=58, srv104331589=5, srv882758420=96, srv1257877016=20, srv1058512945=8, srv524095396=79, srv174360105=41, srv12666098=21, srv1154416672=12, srv758330661=92, srv488988563=77, srv1724264016=40, srv466823732=74, srv1718468310=39, srv1021655348=2, srv1861625149=44, srv1308656614=25, srv1040364476=4, srv317271729=63, srv37593444=68, srv1080731608=9, srv1925424017=49, srv1901597236=46, srv1391596147=28, srv1365571503=27, srv2053851569=54, srv1560296651=31, srv38084764=69, srv1020808623=1, srv1214494252=16, srv1568768539=32, srv164906284=35, srv432673553=72, srv1226926686=18, srv2094503690=56, srv1278220565=22, srv1134008437=11, srv31909970=64, srv1635166200=34, srv1506717936=30, srv1618145457=33, srv739731420=91, srv787214682=94, srv1052351169=6, srv958138367=98, srv1775868100=43, srv838756306=95, srv1704524347=37, srv613899165=83, srv283734874=62, srv142395237=29, srv1289793660=23, srv208511088=55, srv2031061527=53, srv1031476304=3, srv215809107=57, srv118764955=15, srv654986258=87, srv1338151219=26, srv1921946002=48, srv708549152=90, srv1669268617=36, srv1908990207=47, srv763323298=93, srv2015873695=52, srv372256797=67, srv521501575=78, srv105269263=7, srv1171188585=14, srv266237744=60, srv1297866854=24, srv1167114386=13, srv57669160=81, srv1016087569=0, srv1244873975=19, srv283423249=61, srv680324397=88, srv1710294692=38, srv619157000=85, srv652539146=86, srv1876892606=45, srv1954480221=51, srv345335834=66, srv58297225=82, srv1937662969=50, srv1746208879=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:30,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:30,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-11 16:53:30,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-11 16:53:30,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-11 16:53:30,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-11 16:53:30,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-11 16:53:30,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-11 16:53:30,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-11 16:53:30,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-11 16:53:30,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-11 16:53:30,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-11 16:53:30,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-11 16:53:30,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-11 16:53:30,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-11 16:53:30,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:30,260 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1167114386:83 , srv1214494252:83 , srv1221210998:83 , srv1568768539:83 , srv466823732:83 , srv576516028:83 , srv882758420:83 , srv958138367:83 , srv1016087569:84 , srv1020808623:84 , srv1021655348:84 , srv1031476304:84 , srv1040364476:84 , srv104331589:84 , srv1052351169:84 , srv1058512945:84 , srv1080731608:84 , srv1125585439:84 , srv1134008437:84 , srv1154416672:84 , srv1171188585:84 , srv118764955:84 , srv1226926686:84 , srv1244873975:84 , srv1257877016:84 , srv12666098:84 , srv1278220565:84 , srv1297866854:84 , srv1308656614:84 , srv1338151219:84 , srv1365571503:84 , srv1391596147:84 , srv142395237:84 , srv1506717936:84 , srv1560296651:84 , srv1618145457:84 , srv1635166200:84 , srv1669268617:84 , srv1704524347:84 , srv1710294692:84 , srv1724264016:84 , srv174360105:84 , srv1746208879:84 , srv1775868100:84 , srv1861625149:84 , srv1876892606:84 , srv1901597236:84 , srv1908990207:84 , srv1921946002:84 , srv1925424017:84 , srv1937662969:84 , srv1954480221:84 , srv2015873695:84 , srv2031061527:84 , srv2094503690:84 , srv215809107:84 , srv241385811:84 , srv250599473:84 , srv266237744:84 , srv283423249:84 , srv283734874:84 , srv31909970:84 , srv320487659:84 , srv345335834:84 , srv372256797:84 , srv37593444:84 , srv38084764:84 , srv395819388:84 , srv416564834:84 , srv432673553:84 , srv465140754:84 , srv474948006:84 , srv486153366:84 , srv488988563:84 , srv521501575:84 , srv524095396:84 , srv57669160:84 , srv58297225:84 , srv613899165:84 , srv614938477:84 , srv619157000:84 , srv652539146:84 , srv680324397:84 , srv690688190:84 , srv708549152:84 , srv739731420:84 , srv758330661:84 , srv763323298:84 , srv787214682:84 , srv838756306:84 , srv942337035:84 , srv105269263:85 , srv1289793660:85 , srv164906284:85 , srv2053851569:85 , srv208511088:85 , srv317271729:85 , srv654986258:85 , srv983810911:85 , srv1718468310:684 } 2023-10-11 16:53:30,290 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=347 (was 334) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=6625 (was 6637) 2023-10-11 16:53:30,299 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=347, ProcessCount=171, AvailableMemoryMB=6625 2023-10-11 16:53:30,299 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-11 16:53:30,300 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-11 16:53:30,312 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1145049157:84 , srv1305559089:84 , srv1327930197:84 , srv1408309309:84 , srv141354114:84 , srv1541109910:84 , srv1627150200:84 , srv1861834704:84 , srv1917237655:84 , srv1969296736:84 , srv2035653405:84 , srv2052948505:84 , srv2084629377:84 , srv234630540:84 , srv245171420:84 , srv255574777:84 , srv284032902:84 , srv298569443:84 , srv33822379:84 , srv405308367:84 , srv602841961:84 , srv635520638:84 , srv693445937:84 , srv69979939:84 , srv707022229:84 , srv813206612:84 , srv843642328:84 , srv866773668:84 , srv964319864:84 , srv35576567:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-11 16:53:30,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:53:30,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035653405=10, srv1145049157=0, srv964319864=29, srv843642328=27, srv1408309309=3, srv284032902=16, srv707022229=25, srv298569443=17, srv245171420=14, srv813206612=26, srv602841961=21, srv693445937=23, srv866773668=28, srv141354114=4, srv35576567=19, srv33822379=18, srv1861834704=7, srv1541109910=5, srv2052948505=11, srv2084629377=12, srv1969296736=9, srv255574777=15, srv405308367=20, srv635520638=22, srv1627150200=6, srv1327930197=2, srv69979939=24, srv1305559089=1, srv1917237655=8, srv234630540=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-11 16:53:30,357 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0967671549680653, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:53:30,357 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912497953409847, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0967671549680653, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-11 16:55:30,336 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5447851 different iterations. Found a solution that moves 1844 regions; Going from a computed imbalance of 0.9912497953409847 to a new imbalance of 4.303987990979898E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.682962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:55:30,341 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1145049157:90 , srv1305559089:90 , srv1327930197:90 , srv1408309309:90 , srv141354114:90 , srv1541109910:90 , srv1627150200:90 , srv1861834704:90 , srv1917237655:90 , srv1969296736:90 , srv2035653405:90 , srv2052948505:90 , srv2084629377:90 , srv234630540:90 , srv245171420:90 , srv255574777:90 , srv284032902:90 , srv298569443:90 , srv33822379:90 , srv35576567:90 , srv405308367:90 , srv602841961:90 , srv635520638:90 , srv693445937:90 , srv69979939:90 , srv707022229:90 , srv813206612:90 , srv843642328:90 , srv866773668:90 , srv964319864:90 } 2023-10-11 16:55:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-11 16:55:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035653405=10, srv1145049157=0, srv964319864=29, srv843642328=27, srv1408309309=3, srv284032902=16, srv707022229=25, srv298569443=17, srv245171420=14, srv813206612=26, srv602841961=21, srv693445937=23, srv866773668=28, srv141354114=4, srv35576567=19, srv33822379=18, srv1861834704=7, srv1541109910=5, srv2052948505=11, srv2084629377=12, srv1969296736=9, srv255574777=15, srv405308367=20, srv635520638=22, srv1627150200=6, srv1327930197=2, srv69979939=24, srv1305559089=1, srv1917237655=8, srv234630540=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-11 16:55:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-11 16:55:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-11 16:55:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-11 16:55:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-11 16:55:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-11 16:55:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-11 16:55:30,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-11 16:55:30,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-11 16:55:30,359 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-11 16:55:30,359 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1145049157:90 , srv1305559089:90 , srv1327930197:90 , srv1408309309:90 , srv141354114:90 , srv1541109910:90 , srv1627150200:90 , srv1861834704:90 , srv1917237655:90 , srv1969296736:90 , srv2035653405:90 , srv2052948505:90 , srv2084629377:90 , srv234630540:90 , srv245171420:90 , srv255574777:90 , srv284032902:90 , srv298569443:90 , srv33822379:90 , srv35576567:90 , srv405308367:90 , srv602841961:90 , srv635520638:90 , srv693445937:90 , srv69979939:90 , srv707022229:90 , srv813206612:90 , srv843642328:90 , srv866773668:90 , srv964319864:90 } 2023-10-11 16:55:30,372 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 347), ProcessCount=171 (was 171), AvailableMemoryMB=5770 (was 6625)