2024-04-18 15:53:43,926 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-18 15:53:44,292 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-18 15:53:44,446 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-18 15:53:44,519 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-18 15:53:44,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-18 15:53:44,552 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=186, AvailableMemoryMB=6340 2024-04-18 15:53:44,554 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-18 15:53:44,555 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-18 15:53:44,731 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008828506:84 , srv1025709473:84 , srv1045453684:84 , srv1048235726:84 , srv1053996392:84 , srv1070643586:84 , srv1101431284:84 , srv1177859479:84 , srv1195361470:84 , srv1217168008:84 , srv1266547027:84 , srv1268844935:84 , srv1280230589:84 , srv1286134519:84 , srv1296497433:84 , srv1300860294:84 , srv1319113104:84 , srv1321262007:84 , srv1324369370:84 , srv1332160666:84 , srv1333757289:84 , srv1348689683:84 , srv1383714305:84 , srv1396768515:84 , srv1404786321:84 , srv1446727705:84 , srv145283067:84 , srv1458788783:84 , srv1461501137:84 , srv1476249716:84 , srv1482240635:84 , srv1507386047:84 , srv1512432926:84 , srv1517266899:84 , srv1528415310:84 , srv152878240:84 , srv1547966809:84 , srv1562182554:84 , srv1564379260:84 , srv1573955749:84 , srv157839802:84 , srv1581751905:84 , srv159924882:84 , srv1614448342:84 , srv1623339608:84 , srv1652162106:84 , srv1679069933:84 , srv1681517832:84 , srv1723444064:84 , srv1728721267:84 , srv1737075419:84 , srv1738681635:84 , srv1745301890:84 , srv1760646986:84 , srv1762111912:84 , srv1793937665:84 , srv1806341710:84 , srv1810086047:84 , srv1860371767:84 , srv1909198059:84 , srv1911117909:84 , srv1941125979:84 , srv1944159132:84 , srv1971427146:84 , srv1990258365:84 , srv2118212880:84 , srv213697258:84 , srv24181824:84 , srv320153550:84 , srv346817734:84 , srv39146467:84 , srv43259200:84 , srv443786990:84 , srv494243864:84 , srv547626377:84 , srv578194579:84 , srv63456603:84 , srv63768305:84 , srv656650873:84 , srv659101819:84 , srv665465930:84 , srv671625809:84 , srv699037457:84 , srv699072353:84 , srv715331699:84 , srv758792086:84 , srv761934890:84 , srv795310976:84 , srv796315340:84 , srv827374764:84 , srv84282489:84 , srv852200468:84 , srv870486488:84 , srv88877453:84 , srv909504005:84 , srv935276721:84 , srv935919202:84 , srv949144720:84 , srv977920981:84 , srv1019154349:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-18 15:53:44,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:44,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1101431284=7, srv1458788783=28, srv1177859479=8, srv1614448342=44, srv1728721267=50, srv870486488=93, srv1268844935=12, srv346817734=70, srv1333757289=21, srv852200468=92, srv1528415310=35, srv699037457=83, srv1404786321=25, srv1482240635=31, srv1737075419=51, srv1723444064=49, srv1517266899=34, srv1745301890=53, srv1562182554=38, srv43259200=72, srv935276721=96, srv157839802=41, srv699072353=84, srv1652162106=46, srv1681517832=48, srv159924882=43, srv1286134519=14, srv1507386047=32, srv1396768515=24, srv1911117909=61, srv715331699=85, srv1266547027=11, srv1319113104=17, srv1581751905=42, srv1860371767=59, srv1008828506=0, srv1461501137=29, srv758792086=86, srv909504005=95, srv1383714305=23, srv656650873=79, srv1053996392=5, srv1476249716=30, srv1025709473=2, srv1909198059=60, srv84282489=91, srv671625809=82, srv145283067=27, srv1762111912=55, srv827374764=90, srv213697258=67, srv1623339608=45, srv24181824=68, srv494243864=74, srv795310976=88, srv1564379260=39, srv1048235726=4, srv320153550=69, srv1324369370=19, srv1547966809=37, srv1990258365=65, srv63456603=77, srv1019154349=1, srv1738681635=52, srv935919202=97, srv1280230589=13, srv1806341710=57, srv1971427146=64, srv547626377=75, srv1944159132=63, srv1217168008=10, srv1941125979=62, srv152878240=36, srv1760646986=54, srv2118212880=66, srv1793937665=56, srv761934890=87, srv1679069933=47, srv1446727705=26, srv949144720=98, srv1810086047=58, srv665465930=81, srv1512432926=33, srv578194579=76, srv1296497433=15, srv796315340=89, srv39146467=71, srv63768305=78, srv1573955749=40, srv659101819=80, srv1045453684=3, srv443786990=73, srv1321262007=18, srv1300860294=16, srv1070643586=6, srv1332160666=20, srv977920981=99, srv1348689683=22, srv88877453=94, srv1195361470=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:44,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:44,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:44,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:44,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:44,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:44,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:44,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:44,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:44,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:44,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:44,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:44,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:44,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:44,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:44,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:44,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:44,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:44,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:44,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:44,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:44,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:44,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:44,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:44,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:44,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-18 15:53:44,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-18 15:53:44,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-18 15:53:44,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-18 15:53:44,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-18 15:53:44,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-18 15:53:44,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-18 15:53:44,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-18 15:53:44,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-18 15:53:44,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-18 15:53:44,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-18 15:53:44,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-18 15:53:44,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-18 15:53:44,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-18 15:53:44,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-18 15:53:44,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-18 15:53:44,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-18 15:53:44,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-18 15:53:44,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-18 15:53:44,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-18 15:53:44,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-18 15:53:44,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-18 15:53:44,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-18 15:53:44,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-18 15:53:44,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-18 15:53:44,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-18 15:53:44,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-18 15:53:44,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-18 15:53:44,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-18 15:53:44,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-18 15:53:44,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-18 15:53:44,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-18 15:53:44,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-18 15:53:44,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-18 15:53:44,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-18 15:53:44,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-18 15:53:44,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-18 15:53:44,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-18 15:53:44,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-18 15:53:44,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-18 15:53:44,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-18 15:53:44,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-18 15:53:44,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-18 15:53:44,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-18 15:53:44,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-18 15:53:44,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-18 15:53:44,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-18 15:53:44,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-18 15:53:44,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-18 15:53:44,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-18 15:53:44,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-18 15:53:44,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-18 15:53:44,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-18 15:53:45,118 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:45,119 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-18 15:53:45,119 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-18 15:53:45,629 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 697 ms to try 5000 different iterations. Found a solution that moves 4689 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1820433425276663. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039973179685276354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.049924746961271874, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.521, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039973179685276354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17725, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:45,671 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv761934890:80 , srv213697258:81 , srv1101431284:82 , srv1564379260:82 , srv1745301890:82 , srv1911117909:82 , srv1008828506:83 , srv1025709473:83 , srv1333757289:83 , srv1507386047:83 , srv24181824:83 , srv671625809:83 , srv758792086:83 , srv1266547027:84 , srv1300860294:84 , srv1321262007:84 , srv1324369370:84 , srv1348689683:84 , srv1458788783:84 , srv1737075419:84 , srv935276721:84 , srv1177859479:85 , srv1286134519:85 , srv1319113104:85 , srv159924882:85 , srv1681517832:85 , srv1738681635:85 , srv1971427146:85 , srv43259200:85 , srv494243864:85 , srv547626377:85 , srv935919202:85 , srv1045453684:86 , srv1048235726:86 , srv1296497433:86 , srv1332160666:86 , srv1383714305:86 , srv1404786321:86 , srv1446727705:86 , srv145283067:86 , srv1476249716:86 , srv1517266899:86 , srv1528415310:86 , srv152878240:86 , srv1562182554:86 , srv1581751905:86 , srv1652162106:86 , srv1760646986:86 , srv1793937665:86 , srv1810086047:86 , srv1860371767:86 , srv1909198059:86 , srv1941125979:86 , srv1944159132:86 , srv1990258365:86 , srv443786990:86 , srv665465930:86 , srv699037457:86 , srv852200468:86 , srv949144720:86 , srv977920981:86 , srv1396768515:87 , srv1547966809:87 , srv1573955749:87 , srv1623339608:87 , srv1728721267:87 , srv1762111912:87 , srv320153550:87 , srv346817734:87 , srv578194579:87 , srv63456603:87 , srv63768305:87 , srv656650873:87 , srv827374764:87 , srv84282489:87 , srv88877453:87 , srv909504005:87 , srv1280230589:88 , srv1806341710:88 , srv2118212880:88 , srv659101819:88 , srv699072353:88 , srv1070643586:89 , srv1268844935:89 , srv1512432926:89 , srv1614448342:89 , srv39146467:89 , srv795310976:89 , srv1217168008:90 , srv1482240635:90 , srv157839802:90 , srv1461501137:91 , srv796315340:91 , srv1195361470:92 , srv715331699:93 , srv1053996392:94 , srv1679069933:95 , srv870486488:95 , srv1723444064:97 , srv1019154349:445 } 2024-04-18 15:53:45,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:45,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1101431284=7, srv1458788783=28, srv1177859479=8, srv1614448342=44, srv1728721267=50, srv870486488=93, srv1268844935=12, srv346817734=70, srv1333757289=21, srv852200468=92, srv1528415310=35, srv699037457=83, srv1404786321=25, srv1482240635=31, srv1737075419=51, srv1723444064=49, srv1517266899=34, srv1745301890=53, srv1562182554=38, srv43259200=72, srv935276721=96, srv157839802=41, srv699072353=84, srv1652162106=46, srv1681517832=48, srv159924882=43, srv1286134519=14, srv1507386047=32, srv1396768515=24, srv1911117909=61, srv715331699=85, srv1266547027=11, srv1319113104=17, srv1581751905=42, srv1860371767=59, srv1008828506=0, srv1461501137=29, srv758792086=86, srv909504005=95, srv1383714305=23, srv656650873=79, srv1053996392=5, srv1476249716=30, srv1025709473=2, srv1909198059=60, srv84282489=91, srv671625809=82, srv145283067=27, srv1762111912=55, srv827374764=90, srv213697258=67, srv1623339608=45, srv24181824=68, srv494243864=74, srv795310976=88, srv1564379260=39, srv1048235726=4, srv320153550=69, srv1324369370=19, srv1547966809=37, srv1990258365=65, srv63456603=77, srv1019154349=1, srv1738681635=52, srv935919202=97, srv1280230589=13, srv1806341710=57, srv1971427146=64, srv547626377=75, srv1944159132=63, srv1217168008=10, srv1941125979=62, srv152878240=36, srv1760646986=54, srv2118212880=66, srv1793937665=56, srv761934890=87, srv1679069933=47, srv1446727705=26, srv949144720=98, srv1810086047=58, srv665465930=81, srv1512432926=33, srv578194579=76, srv1296497433=15, srv796315340=89, srv39146467=71, srv63768305=78, srv1573955749=40, srv659101819=80, srv1045453684=3, srv443786990=73, srv1321262007=18, srv1300860294=16, srv1070643586=6, srv1332160666=20, srv977920981=99, srv1348689683=22, srv88877453=94, srv1195361470=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:45,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:45,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:45,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-18 15:53:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-18 15:53:45,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-18 15:53:45,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-18 15:53:45,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-18 15:53:45,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-18 15:53:45,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-18 15:53:45,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-18 15:53:45,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-18 15:53:45,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-18 15:53:45,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-18 15:53:45,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-18 15:53:46,074 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039973179685276354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.049924746961271874, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039973179685276354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17725, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:46,074 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-18 15:53:46,074 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18201050944791083, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039973179685276354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.049924746961271874, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039973179685276354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17725, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-18 15:53:46,714 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 887 ms to try 5000 different iterations. Found a solution that moves 2777 regions; Going from a computed imbalance of 0.18201050944791083 to a new imbalance of 0.014742532475725849. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020942977469189328); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030050462609256166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.3085555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020942977469189328); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:46,770 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1723444064:74 , srv1679069933:77 , srv870486488:77 , srv1053996392:78 , srv1195361470:79 , srv1461501137:80 , srv715331699:80 , srv796315340:81 , srv1806341710:82 , srv2118212880:82 , srv346817734:82 , srv659101819:82 , srv699072353:82 , srv1268844935:83 , srv157839802:83 , srv795310976:83 , srv84282489:83 , srv1045453684:84 , srv1070643586:84 , srv1217168008:84 , srv1396768515:84 , srv1482240635:84 , srv1547966809:84 , srv1614448342:84 , srv63768305:84 , srv1280230589:85 , srv1319113104:85 , srv1476249716:85 , srv1512432926:85 , srv1581751905:85 , srv1623339608:85 , srv1971427146:85 , srv320153550:85 , srv443786990:85 , srv578194579:85 , srv63456603:85 , srv88877453:85 , srv909504005:85 , srv935276721:85 , srv949144720:85 , srv977920981:85 , srv1286134519:86 , srv1324369370:86 , srv1348689683:86 , srv1404786321:86 , srv1517266899:86 , srv152878240:86 , srv1573955749:86 , srv1652162106:86 , srv1728721267:86 , srv1738681635:86 , srv1762111912:86 , srv1793937665:86 , srv1810086047:86 , srv1909198059:86 , srv1990258365:86 , srv43259200:86 , srv494243864:86 , srv656650873:86 , srv665465930:86 , srv699037457:86 , srv935919202:86 , srv1048235726:87 , srv1296497433:87 , srv1321262007:87 , srv1332160666:87 , srv145283067:87 , srv159924882:87 , srv1760646986:87 , srv1860371767:87 , srv1941125979:87 , srv1944159132:87 , srv24181824:87 , srv39146467:87 , srv827374764:87 , srv1025709473:88 , srv1300860294:88 , srv1383714305:88 , srv1458788783:88 , srv1528415310:88 , srv1562182554:88 , srv1681517832:88 , srv547626377:88 , srv852200468:88 , srv1101431284:89 , srv1177859479:89 , srv1266547027:89 , srv1737075419:89 , srv1333757289:90 , srv1507386047:90 , srv1745301890:90 , srv1911117909:90 , srv671625809:90 , srv758792086:90 , srv1008828506:91 , srv1564379260:91 , srv213697258:91 , srv761934890:93 , srv1446727705:94 , srv1019154349:515 } 2024-04-18 15:53:46,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:46,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1101431284=7, srv1458788783=28, srv1177859479=8, srv1614448342=44, srv1728721267=50, srv870486488=93, srv1268844935=12, srv346817734=70, srv1333757289=21, srv852200468=92, srv1528415310=35, srv699037457=83, srv1404786321=25, srv1482240635=31, srv1737075419=51, srv1723444064=49, srv1517266899=34, srv1745301890=53, srv1562182554=38, srv43259200=72, srv935276721=96, srv157839802=41, srv699072353=84, srv1652162106=46, srv1681517832=48, srv159924882=43, srv1286134519=14, srv1507386047=32, srv1396768515=24, srv1911117909=61, srv715331699=85, srv1266547027=11, srv1319113104=17, srv1581751905=42, srv1860371767=59, srv1008828506=0, srv1461501137=29, srv758792086=86, srv909504005=95, srv1383714305=23, srv656650873=79, srv1053996392=5, srv1476249716=30, srv1025709473=2, srv1909198059=60, srv84282489=91, srv671625809=82, srv145283067=27, srv1762111912=55, srv827374764=90, srv213697258=67, srv1623339608=45, srv24181824=68, srv494243864=74, srv795310976=88, srv1564379260=39, srv1048235726=4, srv320153550=69, srv1324369370=19, srv1547966809=37, srv1990258365=65, srv63456603=77, srv1019154349=1, srv1738681635=52, srv935919202=97, srv1280230589=13, srv1806341710=57, srv1971427146=64, srv547626377=75, srv1944159132=63, srv1217168008=10, srv1941125979=62, srv152878240=36, srv1760646986=54, srv2118212880=66, srv1793937665=56, srv761934890=87, srv1679069933=47, srv1446727705=26, srv949144720=98, srv1810086047=58, srv665465930=81, srv1512432926=33, srv578194579=76, srv1296497433=15, srv796315340=89, srv39146467=71, srv63768305=78, srv1573955749=40, srv659101819=80, srv1045453684=3, srv443786990=73, srv1321262007=18, srv1300860294=16, srv1070643586=6, srv1332160666=20, srv977920981=99, srv1348689683=22, srv88877453=94, srv1195361470=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:46,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:46,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:46,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:46,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:46,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:46,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:46,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:46,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:46,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:46,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:46,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:46,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:46,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:46,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:46,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:46,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-18 15:53:46,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-18 15:53:46,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-18 15:53:46,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-18 15:53:46,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-18 15:53:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-18 15:53:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-18 15:53:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-18 15:53:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-18 15:53:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-18 15:53:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-18 15:53:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-18 15:53:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-18 15:53:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-18 15:53:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-18 15:53:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-18 15:53:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-18 15:53:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-18 15:53:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-18 15:53:47,200 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020942977469189328); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030050462609256166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020942977469189328); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:47,201 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-18 15:53:47,201 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014723087506120178, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020942977469189328); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030050462609256166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020942977469189328); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-18 15:53:47,904 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 939 ms to try 5000 different iterations. Found a solution that moves 977 regions; Going from a computed imbalance of 0.014723087506120178 to a new imbalance of 4.822303140939638E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005381099233077657); MoveCostFunction : (multiplier=7.0, imbalance=0.10855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006444096124338012); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:47,913 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1446727705:77 , srv39146467:82 , srv1177859479:83 , srv1280230589:83 , srv1383714305:83 , srv1482240635:83 , srv1512432926:83 , srv1528415310:83 , srv852200468:83 , srv1048235726:84 , srv1070643586:84 , srv1217168008:84 , srv1266547027:84 , srv1332160666:84 , srv1333757289:84 , srv1507386047:84 , srv152878240:84 , srv1562182554:84 , srv1573955749:84 , srv1614448342:84 , srv1652162106:84 , srv1681517832:84 , srv1728721267:84 , srv1737075419:84 , srv1860371767:84 , srv1941125979:84 , srv1944159132:84 , srv547626377:84 , srv656650873:84 , srv671625809:84 , srv715331699:84 , srv761934890:84 , srv827374764:84 , srv1008828506:85 , srv1296497433:85 , srv1404786321:85 , srv145283067:85 , srv1458788783:85 , srv1517266899:85 , srv1564379260:85 , srv157839802:85 , srv159924882:85 , srv1760646986:85 , srv1762111912:85 , srv1810086047:85 , srv1911117909:85 , srv63456603:85 , srv63768305:85 , srv665465930:85 , srv758792086:85 , srv795310976:85 , srv870486488:85 , srv88877453:85 , srv909504005:85 , srv1025709473:86 , srv1101431284:86 , srv1268844935:86 , srv1286134519:86 , srv1300860294:86 , srv1461501137:86 , srv1547966809:86 , srv1581751905:86 , srv1623339608:86 , srv1679069933:86 , srv1723444064:86 , srv1745301890:86 , srv1909198059:86 , srv1990258365:86 , srv213697258:86 , srv320153550:86 , srv43259200:86 , srv443786990:86 , srv578194579:86 , srv699037457:86 , srv796315340:86 , srv935919202:86 , srv949144720:86 , srv1045453684:87 , srv1053996392:87 , srv1321262007:87 , srv1324369370:87 , srv1348689683:87 , srv1396768515:87 , srv1476249716:87 , srv1793937665:87 , srv494243864:87 , srv659101819:87 , srv84282489:87 , srv977920981:87 , srv1738681635:88 , srv1806341710:88 , srv1971427146:88 , srv2118212880:88 , srv24181824:88 , srv346817734:88 , srv699072353:88 , srv935276721:88 , srv1195361470:89 , srv1319113104:89 , srv1019154349:555 } 2024-04-18 15:53:47,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:47,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1101431284=7, srv1458788783=28, srv1177859479=8, srv1614448342=44, srv1728721267=50, srv870486488=93, srv1268844935=12, srv346817734=70, srv1333757289=21, srv852200468=92, srv1528415310=35, srv699037457=83, srv1404786321=25, srv1482240635=31, srv1737075419=51, srv1723444064=49, srv1517266899=34, srv1745301890=53, srv1562182554=38, srv43259200=72, srv935276721=96, srv157839802=41, srv699072353=84, srv1652162106=46, srv1681517832=48, srv159924882=43, srv1286134519=14, srv1507386047=32, srv1396768515=24, srv1911117909=61, srv715331699=85, srv1266547027=11, srv1319113104=17, srv1581751905=42, srv1860371767=59, srv1008828506=0, srv1461501137=29, srv758792086=86, srv909504005=95, srv1383714305=23, srv656650873=79, srv1053996392=5, srv1476249716=30, srv1025709473=2, srv1909198059=60, srv84282489=91, srv671625809=82, srv145283067=27, srv1762111912=55, srv827374764=90, srv213697258=67, srv1623339608=45, srv24181824=68, srv494243864=74, srv795310976=88, srv1564379260=39, srv1048235726=4, srv320153550=69, srv1324369370=19, srv1547966809=37, srv1990258365=65, srv63456603=77, srv1019154349=1, srv1738681635=52, srv935919202=97, srv1280230589=13, srv1806341710=57, srv1971427146=64, srv547626377=75, srv1944159132=63, srv1217168008=10, srv1941125979=62, srv152878240=36, srv1760646986=54, srv2118212880=66, srv1793937665=56, srv761934890=87, srv1679069933=47, srv1446727705=26, srv949144720=98, srv1810086047=58, srv665465930=81, srv1512432926=33, srv578194579=76, srv1296497433=15, srv796315340=89, srv39146467=71, srv63768305=78, srv1573955749=40, srv659101819=80, srv1045453684=3, srv443786990=73, srv1321262007=18, srv1300860294=16, srv1070643586=6, srv1332160666=20, srv977920981=99, srv1348689683=22, srv88877453=94, srv1195361470=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:47,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:47,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:47,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:47,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:47,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:47,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:47,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:47,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:47,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:47,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:47,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:47,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-18 15:53:47,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-18 15:53:47,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-18 15:53:47,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-18 15:53:47,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-18 15:53:47,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-18 15:53:47,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-18 15:53:47,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-18 15:53:47,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-18 15:53:47,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-18 15:53:47,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-18 15:53:47,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-18 15:53:47,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-18 15:53:47,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-18 15:53:47,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-18 15:53:47,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-18 15:53:47,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-18 15:53:47,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-18 15:53:48,128 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005381099233077657); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006444096124338012); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:48,129 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-18 15:53:48,129 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.753892138761969E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005381099233077657); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006444096124338012); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-18 15:53:48,557 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 583 ms to try 5000 different iterations. Found a solution that moves 208 regions; Going from a computed imbalance of 4.753892138761969E-4 to a new imbalance of 2.7143142523633887E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.02311111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:48,558 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1053996392:82 , srv1195361470:82 , srv1319113104:82 , srv1008828506:83 , srv157839802:83 , srv1738681635:83 , srv1793937665:83 , srv1806341710:83 , srv1268844935:84 , srv1296497433:84 , srv1300860294:84 , srv1321262007:84 , srv1404786321:84 , srv145283067:84 , srv1476249716:84 , srv1562182554:84 , srv1623339608:84 , srv1679069933:84 , srv1745301890:84 , srv1760646986:84 , srv1762111912:84 , srv1909198059:84 , srv1911117909:84 , srv1971427146:84 , srv1990258365:84 , srv2118212880:84 , srv213697258:84 , srv24181824:84 , srv320153550:84 , srv39146467:84 , srv494243864:84 , srv578194579:84 , srv699037457:84 , srv699072353:84 , srv715331699:84 , srv758792086:84 , srv796315340:84 , srv827374764:84 , srv935919202:84 , srv977920981:84 , srv1025709473:85 , srv1045453684:85 , srv1048235726:85 , srv1070643586:85 , srv1101431284:85 , srv1177859479:85 , srv1217168008:85 , srv1266547027:85 , srv1286134519:85 , srv1324369370:85 , srv1332160666:85 , srv1333757289:85 , srv1348689683:85 , srv1383714305:85 , srv1396768515:85 , srv1446727705:85 , srv1458788783:85 , srv1461501137:85 , srv1482240635:85 , srv1507386047:85 , srv1512432926:85 , srv1517266899:85 , srv1528415310:85 , srv1547966809:85 , srv1564379260:85 , srv1573955749:85 , srv1581751905:85 , srv159924882:85 , srv1614448342:85 , srv1681517832:85 , srv1723444064:85 , srv1728721267:85 , srv1737075419:85 , srv1810086047:85 , srv1860371767:85 , srv1941125979:85 , srv1944159132:85 , srv346817734:85 , srv43259200:85 , srv443786990:85 , srv63456603:85 , srv656650873:85 , srv659101819:85 , srv665465930:85 , srv671625809:85 , srv761934890:85 , srv795310976:85 , srv84282489:85 , srv852200468:85 , srv88877453:85 , srv909504005:85 , srv1280230589:86 , srv1652162106:86 , srv547626377:86 , srv63768305:86 , srv870486488:86 , srv935276721:86 , srv949144720:86 , srv152878240:87 , srv1019154349:627 } 2024-04-18 15:53:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:48,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1101431284=7, srv1458788783=28, srv1177859479=8, srv1614448342=44, srv1728721267=50, srv870486488=93, srv1268844935=12, srv346817734=70, srv1333757289=21, srv852200468=92, srv1528415310=35, srv699037457=83, srv1404786321=25, srv1482240635=31, srv1737075419=51, srv1723444064=49, srv1517266899=34, srv1745301890=53, srv1562182554=38, srv43259200=72, srv935276721=96, srv157839802=41, srv699072353=84, srv1652162106=46, srv1681517832=48, srv159924882=43, srv1286134519=14, srv1507386047=32, srv1396768515=24, srv1911117909=61, srv715331699=85, srv1266547027=11, srv1319113104=17, srv1581751905=42, srv1860371767=59, srv1008828506=0, srv1461501137=29, srv758792086=86, srv909504005=95, srv1383714305=23, srv656650873=79, srv1053996392=5, srv1476249716=30, srv1025709473=2, srv1909198059=60, srv84282489=91, srv671625809=82, srv145283067=27, srv1762111912=55, srv827374764=90, srv213697258=67, srv1623339608=45, srv24181824=68, srv494243864=74, srv795310976=88, srv1564379260=39, srv1048235726=4, srv320153550=69, srv1324369370=19, srv1547966809=37, srv1990258365=65, srv63456603=77, srv1019154349=1, srv1738681635=52, srv935919202=97, srv1280230589=13, srv1806341710=57, srv1971427146=64, srv547626377=75, srv1944159132=63, srv1217168008=10, srv1941125979=62, srv152878240=36, srv1760646986=54, srv2118212880=66, srv1793937665=56, srv761934890=87, srv1679069933=47, srv1446727705=26, srv949144720=98, srv1810086047=58, srv665465930=81, srv1512432926=33, srv578194579=76, srv1296497433=15, srv796315340=89, srv39146467=71, srv63768305=78, srv1573955749=40, srv659101819=80, srv1045453684=3, srv443786990=73, srv1321262007=18, srv1300860294=16, srv1070643586=6, srv1332160666=20, srv977920981=99, srv1348689683=22, srv88877453=94, srv1195361470=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:48,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:48,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:48,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:48,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-18 15:53:48,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-18 15:53:48,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-18 15:53:48,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-18 15:53:48,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-18 15:53:48,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-18 15:53:48,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-18 15:53:48,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-18 15:53:48,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-18 15:53:48,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-18 15:53:48,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-18 15:53:48,718 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:48,718 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-18 15:53:48,719 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.56866953943652E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-18 15:53:49,017 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 418 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 2.56866953943652E-5 to a new imbalance of 6.926124944990534E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:49,017 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1217168008:83 , srv1396768515:83 , srv152878240:83 , srv1564379260:83 , srv547626377:83 , srv870486488:83 , srv935276721:83 , srv949144720:83 , srv1008828506:84 , srv1025709473:84 , srv1045453684:84 , srv1048235726:84 , srv1070643586:84 , srv1101431284:84 , srv1177859479:84 , srv1195361470:84 , srv1266547027:84 , srv1268844935:84 , srv1280230589:84 , srv1286134519:84 , srv1296497433:84 , srv1300860294:84 , srv1321262007:84 , srv1324369370:84 , srv1332160666:84 , srv1333757289:84 , srv1348689683:84 , srv1383714305:84 , srv1446727705:84 , srv145283067:84 , srv1458788783:84 , srv1461501137:84 , srv1476249716:84 , srv1482240635:84 , srv1507386047:84 , srv1512432926:84 , srv1517266899:84 , srv1528415310:84 , srv1547966809:84 , srv1562182554:84 , srv1573955749:84 , srv1581751905:84 , srv159924882:84 , srv1614448342:84 , srv1623339608:84 , srv1652162106:84 , srv1679069933:84 , srv1681517832:84 , srv1723444064:84 , srv1728721267:84 , srv1737075419:84 , srv1738681635:84 , srv1745301890:84 , srv1760646986:84 , srv1762111912:84 , srv1793937665:84 , srv1810086047:84 , srv1860371767:84 , srv1909198059:84 , srv1941125979:84 , srv1944159132:84 , srv1971427146:84 , srv1990258365:84 , srv2118212880:84 , srv213697258:84 , srv24181824:84 , srv320153550:84 , srv346817734:84 , srv39146467:84 , srv43259200:84 , srv443786990:84 , srv494243864:84 , srv578194579:84 , srv63456603:84 , srv63768305:84 , srv656650873:84 , srv659101819:84 , srv665465930:84 , srv671625809:84 , srv699037457:84 , srv699072353:84 , srv758792086:84 , srv761934890:84 , srv795310976:84 , srv796315340:84 , srv827374764:84 , srv84282489:84 , srv852200468:84 , srv88877453:84 , srv909504005:84 , srv977920981:84 , srv1053996392:85 , srv1319113104:85 , srv1404786321:85 , srv157839802:85 , srv1806341710:85 , srv1911117909:85 , srv715331699:85 , srv935919202:85 , srv1019154349:684 } 2024-04-18 15:53:49,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:49,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1101431284=7, srv1458788783=28, srv1177859479=8, srv1614448342=44, srv1728721267=50, srv870486488=93, srv1268844935=12, srv346817734=70, srv1333757289=21, srv852200468=92, srv1528415310=35, srv699037457=83, srv1404786321=25, srv1482240635=31, srv1737075419=51, srv1723444064=49, srv1517266899=34, srv1745301890=53, srv1562182554=38, srv43259200=72, srv935276721=96, srv157839802=41, srv699072353=84, srv1652162106=46, srv1681517832=48, srv159924882=43, srv1286134519=14, srv1507386047=32, srv1396768515=24, srv1911117909=61, srv715331699=85, srv1266547027=11, srv1319113104=17, srv1581751905=42, srv1860371767=59, srv1008828506=0, srv1461501137=29, srv758792086=86, srv909504005=95, srv1383714305=23, srv656650873=79, srv1053996392=5, srv1476249716=30, srv1025709473=2, srv1909198059=60, srv84282489=91, srv671625809=82, srv145283067=27, srv1762111912=55, srv827374764=90, srv213697258=67, srv1623339608=45, srv24181824=68, srv494243864=74, srv795310976=88, srv1564379260=39, srv1048235726=4, srv320153550=69, srv1324369370=19, srv1547966809=37, srv1990258365=65, srv63456603=77, srv1019154349=1, srv1738681635=52, srv935919202=97, srv1280230589=13, srv1806341710=57, srv1971427146=64, srv547626377=75, srv1944159132=63, srv1217168008=10, srv1941125979=62, srv152878240=36, srv1760646986=54, srv2118212880=66, srv1793937665=56, srv761934890=87, srv1679069933=47, srv1446727705=26, srv949144720=98, srv1810086047=58, srv665465930=81, srv1512432926=33, srv578194579=76, srv1296497433=15, srv796315340=89, srv39146467=71, srv63768305=78, srv1573955749=40, srv659101819=80, srv1045453684=3, srv443786990=73, srv1321262007=18, srv1300860294=16, srv1070643586=6, srv1332160666=20, srv977920981=99, srv1348689683=22, srv88877453=94, srv1195361470=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:49,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:49,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:49,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:49,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-18 15:53:49,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-18 15:53:49,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-18 15:53:49,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-18 15:53:49,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-18 15:53:49,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-18 15:53:49,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-18 15:53:49,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-18 15:53:49,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-18 15:53:49,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-18 15:53:49,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-18 15:53:49,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-18 15:53:49,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-18 15:53:49,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:49,224 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1217168008:83 , srv1396768515:83 , srv152878240:83 , srv1564379260:83 , srv547626377:83 , srv870486488:83 , srv935276721:83 , srv949144720:83 , srv1008828506:84 , srv1025709473:84 , srv1045453684:84 , srv1048235726:84 , srv1070643586:84 , srv1101431284:84 , srv1177859479:84 , srv1195361470:84 , srv1266547027:84 , srv1268844935:84 , srv1280230589:84 , srv1286134519:84 , srv1296497433:84 , srv1300860294:84 , srv1321262007:84 , srv1324369370:84 , srv1332160666:84 , srv1333757289:84 , srv1348689683:84 , srv1383714305:84 , srv1446727705:84 , srv145283067:84 , srv1458788783:84 , srv1461501137:84 , srv1476249716:84 , srv1482240635:84 , srv1507386047:84 , srv1512432926:84 , srv1517266899:84 , srv1528415310:84 , srv1547966809:84 , srv1562182554:84 , srv1573955749:84 , srv1581751905:84 , srv159924882:84 , srv1614448342:84 , srv1623339608:84 , srv1652162106:84 , srv1679069933:84 , srv1681517832:84 , srv1723444064:84 , srv1728721267:84 , srv1737075419:84 , srv1738681635:84 , srv1745301890:84 , srv1760646986:84 , srv1762111912:84 , srv1793937665:84 , srv1810086047:84 , srv1860371767:84 , srv1909198059:84 , srv1941125979:84 , srv1944159132:84 , srv1971427146:84 , srv1990258365:84 , srv2118212880:84 , srv213697258:84 , srv24181824:84 , srv320153550:84 , srv346817734:84 , srv39146467:84 , srv43259200:84 , srv443786990:84 , srv494243864:84 , srv578194579:84 , srv63456603:84 , srv63768305:84 , srv656650873:84 , srv659101819:84 , srv665465930:84 , srv671625809:84 , srv699037457:84 , srv699072353:84 , srv758792086:84 , srv761934890:84 , srv795310976:84 , srv796315340:84 , srv827374764:84 , srv84282489:84 , srv852200468:84 , srv88877453:84 , srv909504005:84 , srv977920981:84 , srv1053996392:85 , srv1319113104:85 , srv1404786321:85 , srv157839802:85 , srv1806341710:85 , srv1911117909:85 , srv715331699:85 , srv935919202:85 , srv1019154349:684 } 2024-04-18 15:53:49,275 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=730 (was 689) - SystemLoadAverage LEAK? -, ProcessCount=186 (was 186), AvailableMemoryMB=5523 (was 6340) 2024-04-18 15:53:49,291 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=730, ProcessCount=186, AvailableMemoryMB=5522 2024-04-18 15:53:49,291 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-18 15:53:49,292 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-18 15:53:49,312 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018927698:84 , srv1122225264:84 , srv1160317839:84 , srv116473087:84 , srv12593151:84 , srv1446241222:84 , srv1469252141:84 , srv1504483006:84 , srv1555059648:84 , srv1734125371:84 , srv1905777234:84 , srv1915748370:84 , srv1995008850:84 , srv1997558799:84 , srv1998146414:84 , srv2004546215:84 , srv2078107609:84 , srv2124595678:84 , srv221855698:84 , srv227756821:84 , srv260106884:84 , srv38894151:84 , srv693221989:84 , srv777598249:84 , srv785871094:84 , srv872482603:84 , srv884142003:84 , srv918906437:84 , srv926826464:84 , srv1864511936:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-18 15:53:49,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:53:49,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1997558799=14, srv1998146414=15, srv116473087=3, srv1504483006=7, srv38894151=22, srv2004546215=16, srv1905777234=11, srv1018927698=0, srv12593151=4, srv884142003=27, srv260106884=21, srv227756821=20, srv1122225264=1, srv1446241222=5, srv785871094=25, srv777598249=24, srv2124595678=18, srv1864511936=10, srv1734125371=9, srv221855698=19, srv926826464=29, srv1160317839=2, srv1469252141=6, srv1555059648=8, srv2078107609=17, srv1915748370=12, srv918906437=28, srv1995008850=13, srv693221989=23, srv872482603=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:53:49,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:53:49,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:53:49,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-18 15:53:49,351 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1668169908514536, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:53:49,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912718678155376, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1668169908514536, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-18 15:55:49,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 4850349 different iterations. Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912718678155376 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:55:49,341 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018927698:90 , srv1122225264:90 , srv1160317839:90 , srv116473087:90 , srv12593151:90 , srv1446241222:90 , srv1469252141:90 , srv1504483006:90 , srv1555059648:90 , srv1734125371:90 , srv1864511936:90 , srv1905777234:90 , srv1915748370:90 , srv1995008850:90 , srv1997558799:90 , srv1998146414:90 , srv2004546215:90 , srv2078107609:90 , srv2124595678:90 , srv221855698:90 , srv227756821:90 , srv260106884:90 , srv38894151:90 , srv693221989:90 , srv777598249:90 , srv785871094:90 , srv872482603:90 , srv884142003:90 , srv918906437:90 , srv926826464:90 } 2024-04-18 15:55:49,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-18 15:55:49,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1997558799=14, srv1998146414=15, srv116473087=3, srv1504483006=7, srv38894151=22, srv2004546215=16, srv1905777234=11, srv1018927698=0, srv12593151=4, srv884142003=27, srv260106884=21, srv227756821=20, srv1122225264=1, srv1446241222=5, srv785871094=25, srv777598249=24, srv2124595678=18, srv1864511936=10, srv1734125371=9, srv221855698=19, srv926826464=29, srv1160317839=2, srv1469252141=6, srv1555059648=8, srv2078107609=17, srv1915748370=12, srv918906437=28, srv1995008850=13, srv693221989=23, srv872482603=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-18 15:55:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-18 15:55:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-18 15:55:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-18 15:55:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-18 15:55:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-18 15:55:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-18 15:55:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-18 15:55:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-18 15:55:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-18 15:55:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-18 15:55:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-18 15:55:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-18 15:55:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-18 15:55:49,363 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-18 15:55:49,363 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1018927698:90 , srv1122225264:90 , srv1160317839:90 , srv116473087:90 , srv12593151:90 , srv1446241222:90 , srv1469252141:90 , srv1504483006:90 , srv1555059648:90 , srv1734125371:90 , srv1864511936:90 , srv1905777234:90 , srv1915748370:90 , srv1995008850:90 , srv1997558799:90 , srv1998146414:90 , srv2004546215:90 , srv2078107609:90 , srv2124595678:90 , srv221855698:90 , srv227756821:90 , srv260106884:90 , srv38894151:90 , srv693221989:90 , srv777598249:90 , srv785871094:90 , srv872482603:90 , srv884142003:90 , srv918906437:90 , srv926826464:90 } 2024-04-18 15:55:49,384 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=479 (was 730), ProcessCount=185 (was 186), AvailableMemoryMB=4425 (was 5522)