2023-09-01 12:58:49,922 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-01 12:58:50,189 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-01 12:58:50,290 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-01 12:58:50,352 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-01 12:58:50,360 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-01 12:58:50,375 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=257, ProcessCount=166, AvailableMemoryMB=6038 2023-09-01 12:58:50,380 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-01 12:58:50,380 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-01 12:58:50,527 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015233269:84 , srv1025831096:84 , srv1099812634:84 , srv1109273719:84 , srv1124456074:84 , srv112699978:84 , srv1132490800:84 , srv1167692866:84 , srv1168181125:84 , srv1226322429:84 , srv1229931360:84 , srv1238458761:84 , srv1245360656:84 , srv1289054515:84 , srv1316756146:84 , srv1333079644:84 , srv1410174819:84 , srv1417345652:84 , srv142132921:84 , srv142200861:84 , srv1465452434:84 , srv1467441778:84 , srv148561286:84 , srv1502924087:84 , srv1510149608:84 , srv151066670:84 , srv152336277:84 , srv1536020236:84 , srv1536762713:84 , srv1544251915:84 , srv1554886195:84 , srv1556507361:84 , srv1566881129:84 , srv1567381724:84 , srv1626630171:84 , srv1641359002:84 , srv1641429558:84 , srv1642939663:84 , srv1660995637:84 , srv1662818069:84 , srv1677231915:84 , srv1684632003:84 , srv1691854898:84 , srv1697694710:84 , srv1703896406:84 , srv1727500160:84 , srv1728157620:84 , srv177487662:84 , srv1789274169:84 , srv1827974742:84 , srv1832764600:84 , srv1888148249:84 , srv1891071020:84 , srv1892675751:84 , srv1918913273:84 , srv1953418212:84 , srv1969931799:84 , srv2006088135:84 , srv2031688399:84 , srv2113366633:84 , srv2127034933:84 , srv2130760530:84 , srv213743016:84 , srv2138150939:84 , srv221640249:84 , srv223272156:84 , srv23296469:84 , srv238145400:84 , srv240565309:84 , srv244506920:84 , srv254885644:84 , srv303428026:84 , srv303537177:84 , srv30473677:84 , srv309582065:84 , srv320919189:84 , srv347633784:84 , srv384670142:84 , srv404937124:84 , srv436685318:84 , srv455830398:84 , srv468237166:84 , srv496619863:84 , srv562961356:84 , srv563662290:84 , srv56512202:84 , srv607837081:84 , srv67611046:84 , srv712596408:84 , srv719246504:84 , srv743654765:84 , srv756400963:84 , srv75645107:84 , srv765663733:84 , srv801993495:84 , srv8402981:84 , srv91941334:84 , srv936429796:84 , srv959747368:84 , srv479063391:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-01 12:58:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv177487662=47, srv1626630171=34, srv1025831096=1, srv563662290=85, srv1641429558=36, srv151066670=25, srv1556507361=31, srv1502924087=23, srv23296469=66, srv303537177=72, srv1238458761=11, srv1691854898=42, srv2127034933=60, srv1124456074=4, srv1891071020=52, srv1417345652=17, srv309582065=74, srv142200861=19, srv468237166=81, srv1333079644=15, srv8402981=96, srv2130760530=61, srv1410174819=16, srv320919189=75, srv719246504=90, srv303428026=71, srv1969931799=56, srv404937124=78, srv436685318=79, srv1554886195=30, srv1245360656=12, srv1641359002=35, srv1660995637=38, srv2006088135=57, srv1789274169=48, srv244506920=69, srv1827974742=49, srv1892675751=53, srv1316756146=14, srv1109273719=3, srv1168181125=8, srv1099812634=2, srv75645107=93, srv240565309=68, srv712596408=89, srv1544251915=29, srv1888148249=51, srv347633784=76, srv1467441778=21, srv2031688399=58, srv1536020236=27, srv1677231915=40, srv1684632003=41, srv1953418212=55, srv223272156=65, srv254885644=70, srv148561286=22, srv959747368=99, srv765663733=94, srv112699978=5, srv1510149608=24, srv455830398=80, srv1727500160=45, srv1728157620=46, srv1226322429=9, srv1289054515=13, srv1167692866=7, srv936429796=98, srv1465452434=20, srv1697694710=43, srv2138150939=63, srv91941334=97, srv30473677=73, srv1642939663=37, srv562961356=84, srv1132490800=6, srv142132921=18, srv56512202=86, srv743654765=91, srv756400963=92, srv496619863=83, srv221640249=64, srv1015233269=0, srv1229931360=10, srv1703896406=44, srv1566881129=32, srv607837081=87, srv238145400=67, srv1918913273=54, srv1567381724=33, srv152336277=26, srv384670142=77, srv67611046=88, srv1832764600=50, srv1536762713=28, srv1662818069=39, srv2113366633=59, srv213743016=62, srv479063391=82, srv801993495=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:50,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:50,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:50,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:50,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:50,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:50,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:50,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:50,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:50,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:50,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:50,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:50,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-01 12:58:50,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-01 12:58:50,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-01 12:58:50,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-01 12:58:50,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-01 12:58:50,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-01 12:58:50,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-01 12:58:50,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-01 12:58:50,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-01 12:58:50,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-01 12:58:50,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-01 12:58:50,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-01 12:58:50,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-01 12:58:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-01 12:58:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-01 12:58:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-01 12:58:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-01 12:58:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-01 12:58:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-01 12:58:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-01 12:58:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-01 12:58:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-01 12:58:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-01 12:58:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-01 12:58:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-01 12:58:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-01 12:58:50,980 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:50,980 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-01 12:58:50,981 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-01 12:58:51,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 554 ms to try 5000 different iterations. Found a solution that moves 4673 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18394073597772762. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04051082969538891, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05588789484426352, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5192222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04051082969538891, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25033333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:51,275 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1566881129:82 , srv1642939663:82 , srv320919189:82 , srv756400963:82 , srv112699978:83 , srv1410174819:83 , srv1544251915:83 , srv1567381724:83 , srv1641429558:83 , srv1697694710:83 , srv1888148249:83 , srv765663733:83 , srv1025831096:84 , srv1124456074:84 , srv1417345652:84 , srv1536762713:84 , srv1691854898:84 , srv1891071020:84 , srv2113366633:84 , srv562961356:84 , srv712596408:84 , srv801993495:84 , srv8402981:84 , srv959747368:84 , srv1167692866:85 , srv1226322429:85 , srv1502924087:85 , srv152336277:85 , srv1536020236:85 , srv1662818069:85 , srv1918913273:85 , srv2006088135:85 , srv2031688399:85 , srv2138150939:85 , srv223272156:85 , srv303428026:85 , srv563662290:85 , srv75645107:85 , srv1168181125:86 , srv1229931360:86 , srv1245360656:86 , srv1289054515:86 , srv1316756146:86 , srv1333079644:86 , srv1465452434:86 , srv148561286:86 , srv1660995637:86 , srv1684632003:86 , srv1827974742:86 , srv1832764600:86 , srv1892675751:86 , srv1953418212:86 , srv303537177:86 , srv455830398:86 , srv743654765:86 , srv1132490800:87 , srv142132921:87 , srv1556507361:87 , srv1641359002:87 , srv1703896406:87 , srv1727500160:87 , srv177487662:87 , srv221640249:87 , srv238145400:87 , srv309582065:87 , srv347633784:87 , srv384670142:87 , srv404937124:87 , srv468237166:87 , srv719246504:87 , srv91941334:87 , srv1015233269:88 , srv1109273719:88 , srv1467441778:88 , srv1510149608:88 , srv151066670:88 , srv1677231915:88 , srv1728157620:88 , srv1789274169:88 , srv1969931799:88 , srv2127034933:88 , srv23296469:88 , srv240565309:88 , srv496619863:88 , srv607837081:88 , srv142200861:89 , srv244506920:89 , srv30473677:89 , srv436685318:89 , srv1238458761:90 , srv1554886195:90 , srv2130760530:90 , srv213743016:90 , srv1626630171:91 , srv254885644:91 , srv56512202:91 , srv67611046:92 , srv936429796:95 , srv1099812634:98 , srv479063391:450 } 2023-09-01 12:58:51,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:51,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv177487662=47, srv1626630171=34, srv1025831096=1, srv563662290=85, srv1641429558=36, srv151066670=25, srv1556507361=31, srv1502924087=23, srv23296469=66, srv303537177=72, srv1238458761=11, srv1691854898=42, srv2127034933=60, srv1124456074=4, srv1891071020=52, srv1417345652=17, srv309582065=74, srv142200861=19, srv468237166=81, srv1333079644=15, srv8402981=96, srv2130760530=61, srv1410174819=16, srv320919189=75, srv719246504=90, srv303428026=71, srv1969931799=56, srv404937124=78, srv436685318=79, srv1554886195=30, srv1245360656=12, srv1641359002=35, srv1660995637=38, srv2006088135=57, srv1789274169=48, srv244506920=69, srv1827974742=49, srv1892675751=53, srv1316756146=14, srv1109273719=3, srv1168181125=8, srv1099812634=2, srv75645107=93, srv240565309=68, srv712596408=89, srv1544251915=29, srv1888148249=51, srv347633784=76, srv1467441778=21, srv2031688399=58, srv1536020236=27, srv1677231915=40, srv1684632003=41, srv1953418212=55, srv223272156=65, srv254885644=70, srv148561286=22, srv959747368=99, srv765663733=94, srv112699978=5, srv1510149608=24, srv455830398=80, srv1727500160=45, srv1728157620=46, srv1226322429=9, srv1289054515=13, srv1167692866=7, srv936429796=98, srv1465452434=20, srv1697694710=43, srv2138150939=63, srv91941334=97, srv30473677=73, srv1642939663=37, srv562961356=84, srv1132490800=6, srv142132921=18, srv56512202=86, srv743654765=91, srv756400963=92, srv496619863=83, srv221640249=64, srv1015233269=0, srv1229931360=10, srv1703896406=44, srv1566881129=32, srv607837081=87, srv238145400=67, srv1918913273=54, srv1567381724=33, srv152336277=26, srv384670142=77, srv67611046=88, srv1832764600=50, srv1536762713=28, srv1662818069=39, srv2113366633=59, srv213743016=62, srv479063391=82, srv801993495=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-01 12:58:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-01 12:58:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-01 12:58:51,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-01 12:58:51,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-01 12:58:51,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-01 12:58:51,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-01 12:58:51,424 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04051082969538891, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05588789484426352, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04051082969538891, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25033333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:51,425 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-01 12:58:51,425 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18390801493236672, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04051082969538891, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05588789484426352, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04051082969538891, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25033333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-01 12:58:51,658 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 330 ms to try 5000 different iterations. Found a solution that moves 2641 regions; Going from a computed imbalance of 0.18390801493236672 to a new imbalance of 0.017499200055870582. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024288349690019518); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03575386615466377, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29344444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024288349690019518); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01425); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:51,668 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1099812634:74 , srv142200861:80 , srv56512202:80 , srv1238458761:81 , srv213743016:81 , srv254885644:81 , srv936429796:81 , srv151066670:82 , srv1554886195:82 , srv1626630171:82 , srv1727500160:82 , srv30473677:82 , srv607837081:82 , srv67611046:82 , srv1109273719:83 , srv2127034933:83 , srv2130760530:83 , srv221640249:83 , srv23296469:83 , srv244506920:83 , srv347633784:83 , srv404937124:83 , srv496619863:83 , srv1015233269:84 , srv1229931360:84 , srv1677231915:84 , srv238145400:84 , srv1245360656:85 , srv142132921:85 , srv1465452434:85 , srv1467441778:85 , srv148561286:85 , srv1510149608:85 , srv152336277:85 , srv1556507361:85 , srv1641359002:85 , srv1703896406:85 , srv177487662:85 , srv1827974742:85 , srv1892675751:85 , srv1969931799:85 , srv309582065:85 , srv384670142:85 , srv436685318:85 , srv743654765:85 , srv91941334:85 , srv1132490800:86 , srv1167692866:86 , srv1168181125:86 , srv1316756146:86 , srv1333079644:86 , srv1660995637:86 , srv1662818069:86 , srv1691854898:86 , srv1728157620:86 , srv1789274169:86 , srv1832764600:86 , srv223272156:86 , srv240565309:86 , srv468237166:86 , srv563662290:86 , srv712596408:86 , srv801993495:86 , srv959747368:86 , srv112699978:87 , srv1289054515:87 , srv1417345652:87 , srv1536020236:87 , srv1641429558:87 , srv1953418212:87 , srv2113366633:87 , srv2138150939:87 , srv303428026:87 , srv303537177:87 , srv1226322429:88 , srv1502924087:88 , srv1544251915:88 , srv1566881129:88 , srv1567381724:88 , srv1684632003:88 , srv1697694710:88 , srv1888148249:88 , srv1891071020:88 , srv1918913273:88 , srv2006088135:88 , srv2031688399:88 , srv455830398:88 , srv562961356:88 , srv719246504:88 , srv765663733:88 , srv1025831096:89 , srv1124456074:89 , srv1410174819:89 , srv1536762713:89 , srv1642939663:89 , srv320919189:89 , srv75645107:89 , srv8402981:90 , srv756400963:91 , srv479063391:540 } 2023-09-01 12:58:51,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:51,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv177487662=47, srv1626630171=34, srv1025831096=1, srv563662290=85, srv1641429558=36, srv151066670=25, srv1556507361=31, srv1502924087=23, srv23296469=66, srv303537177=72, srv1238458761=11, srv1691854898=42, srv2127034933=60, srv1124456074=4, srv1891071020=52, srv1417345652=17, srv309582065=74, srv142200861=19, srv468237166=81, srv1333079644=15, srv8402981=96, srv2130760530=61, srv1410174819=16, srv320919189=75, srv719246504=90, srv303428026=71, srv1969931799=56, srv404937124=78, srv436685318=79, srv1554886195=30, srv1245360656=12, srv1641359002=35, srv1660995637=38, srv2006088135=57, srv1789274169=48, srv244506920=69, srv1827974742=49, srv1892675751=53, srv1316756146=14, srv1109273719=3, srv1168181125=8, srv1099812634=2, srv75645107=93, srv240565309=68, srv712596408=89, srv1544251915=29, srv1888148249=51, srv347633784=76, srv1467441778=21, srv2031688399=58, srv1536020236=27, srv1677231915=40, srv1684632003=41, srv1953418212=55, srv223272156=65, srv254885644=70, srv148561286=22, srv959747368=99, srv765663733=94, srv112699978=5, srv1510149608=24, srv455830398=80, srv1727500160=45, srv1728157620=46, srv1226322429=9, srv1289054515=13, srv1167692866=7, srv936429796=98, srv1465452434=20, srv1697694710=43, srv2138150939=63, srv91941334=97, srv30473677=73, srv1642939663=37, srv562961356=84, srv1132490800=6, srv142132921=18, srv56512202=86, srv743654765=91, srv756400963=92, srv496619863=83, srv221640249=64, srv1015233269=0, srv1229931360=10, srv1703896406=44, srv1566881129=32, srv607837081=87, srv238145400=67, srv1918913273=54, srv1567381724=33, srv152336277=26, srv384670142=77, srv67611046=88, srv1832764600=50, srv1536762713=28, srv1662818069=39, srv2113366633=59, srv213743016=62, srv479063391=82, srv801993495=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:51,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:51,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:51,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:51,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:51,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:51,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:51,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:51,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-01 12:58:51,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-01 12:58:51,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-01 12:58:51,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-01 12:58:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-01 12:58:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-01 12:58:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-01 12:58:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-01 12:58:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-01 12:58:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-01 12:58:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-01 12:58:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-01 12:58:51,829 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024288349690019518); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03575386615466377, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024288349690019518); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01425); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:51,829 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-01 12:58:51,829 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017480707378618666, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024288349690019518); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03575386615466377, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024288349690019518); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01425); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-01 12:58:52,047 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 315 ms to try 5000 different iterations. Found a solution that moves 1043 regions; Going from a computed imbalance of 0.017480707378618666 to a new imbalance of 0.001517713138818829. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008114408259335794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010194516692557684); MoveCostFunction : (multiplier=7.0, imbalance=0.11588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008114408259335794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.006666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,050 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv936429796:81 , srv1025831096:83 , srv1226322429:83 , srv1684632003:83 , srv1728157620:83 , srv303537177:83 , srv67611046:83 , srv719246504:83 , srv75645107:83 , srv1124456074:84 , srv1132490800:84 , srv1316756146:84 , srv142132921:84 , srv1467441778:84 , srv1536762713:84 , srv1789274169:84 , srv1918913273:84 , srv2031688399:84 , srv2130760530:84 , srv2138150939:84 , srv240565309:84 , srv436685318:84 , srv455830398:84 , srv756400963:84 , srv8402981:84 , srv1168181125:85 , srv1289054515:85 , srv1502924087:85 , srv1510149608:85 , srv1556507361:85 , srv1660995637:85 , srv1677231915:85 , srv1697694710:85 , srv1832764600:85 , srv1953418212:85 , srv1969931799:85 , srv2006088135:85 , srv254885644:85 , srv309582065:85 , srv384670142:85 , srv468237166:85 , srv562961356:85 , srv563662290:85 , srv91941334:85 , srv1015233269:86 , srv1099812634:86 , srv1109273719:86 , srv1167692866:86 , srv1238458761:86 , srv1333079644:86 , srv1410174819:86 , srv1417345652:86 , srv1465452434:86 , srv148561286:86 , srv1536020236:86 , srv1554886195:86 , srv1567381724:86 , srv1626630171:86 , srv1641359002:86 , srv1642939663:86 , srv1662818069:86 , srv1691854898:86 , srv177487662:86 , srv1827974742:86 , srv1888148249:86 , srv1891071020:86 , srv1892675751:86 , srv2113366633:86 , srv2127034933:86 , srv223272156:86 , srv238145400:86 , srv244506920:86 , srv303428026:86 , srv30473677:86 , srv320919189:86 , srv496619863:86 , srv56512202:86 , srv112699978:87 , srv1229931360:87 , srv1245360656:87 , srv142200861:87 , srv151066670:87 , srv152336277:87 , srv1544251915:87 , srv1566881129:87 , srv1641429558:87 , srv1703896406:87 , srv213743016:87 , srv221640249:87 , srv23296469:87 , srv404937124:87 , srv607837081:87 , srv712596408:87 , srv743654765:87 , srv765663733:87 , srv1727500160:88 , srv347633784:88 , srv801993495:88 , srv959747368:88 , srv479063391:540 } 2023-09-01 12:58:52,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:52,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv177487662=47, srv1626630171=34, srv1025831096=1, srv563662290=85, srv1641429558=36, srv151066670=25, srv1556507361=31, srv1502924087=23, srv23296469=66, srv303537177=72, srv1238458761=11, srv1691854898=42, srv2127034933=60, srv1124456074=4, srv1891071020=52, srv1417345652=17, srv309582065=74, srv142200861=19, srv468237166=81, srv1333079644=15, srv8402981=96, srv2130760530=61, srv1410174819=16, srv320919189=75, srv719246504=90, srv303428026=71, srv1969931799=56, srv404937124=78, srv436685318=79, srv1554886195=30, srv1245360656=12, srv1641359002=35, srv1660995637=38, srv2006088135=57, srv1789274169=48, srv244506920=69, srv1827974742=49, srv1892675751=53, srv1316756146=14, srv1109273719=3, srv1168181125=8, srv1099812634=2, srv75645107=93, srv240565309=68, srv712596408=89, srv1544251915=29, srv1888148249=51, srv347633784=76, srv1467441778=21, srv2031688399=58, srv1536020236=27, srv1677231915=40, srv1684632003=41, srv1953418212=55, srv223272156=65, srv254885644=70, srv148561286=22, srv959747368=99, srv765663733=94, srv112699978=5, srv1510149608=24, srv455830398=80, srv1727500160=45, srv1728157620=46, srv1226322429=9, srv1289054515=13, srv1167692866=7, srv936429796=98, srv1465452434=20, srv1697694710=43, srv2138150939=63, srv91941334=97, srv30473677=73, srv1642939663=37, srv562961356=84, srv1132490800=6, srv142132921=18, srv56512202=86, srv743654765=91, srv756400963=92, srv496619863=83, srv221640249=64, srv1015233269=0, srv1229931360=10, srv1703896406=44, srv1566881129=32, srv607837081=87, srv238145400=67, srv1918913273=54, srv1567381724=33, srv152336277=26, srv384670142=77, srv67611046=88, srv1832764600=50, srv1536762713=28, srv1662818069=39, srv2113366633=59, srv213743016=62, srv479063391=82, srv801993495=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:52,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:52,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:52,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:52,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:52,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-01 12:58:52,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-01 12:58:52,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-01 12:58:52,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-01 12:58:52,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-01 12:58:52,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-01 12:58:52,157 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008114408259335794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010194516692557684); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008114408259335794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.006666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,157 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-01 12:58:52,158 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0015104098967235057, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008114408259335794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010194516692557684); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008114408259335794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.006666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-01 12:58:52,389 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 304 ms to try 5000 different iterations. Found a solution that moves 301 regions; Going from a computed imbalance of 0.0015104098967235057 to a new imbalance of 1.2564451958202877E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03344444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,390 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1554886195:83 , srv1626630171:83 , srv1015233269:84 , srv1099812634:84 , srv1109273719:84 , srv1132490800:84 , srv1229931360:84 , srv1245360656:84 , srv1289054515:84 , srv1333079644:84 , srv1510149608:84 , srv152336277:84 , srv1544251915:84 , srv1641359002:84 , srv1703896406:84 , srv1728157620:84 , srv177487662:84 , srv1789274169:84 , srv1832764600:84 , srv1891071020:84 , srv1953418212:84 , srv1969931799:84 , srv2006088135:84 , srv213743016:84 , srv23296469:84 , srv240565309:84 , srv244506920:84 , srv303428026:84 , srv30473677:84 , srv309582065:84 , srv347633784:84 , srv436685318:84 , srv468237166:84 , srv712596408:84 , srv719246504:84 , srv743654765:84 , srv765663733:84 , srv801993495:84 , srv8402981:84 , srv936429796:84 , srv959747368:84 , srv1124456074:85 , srv112699978:85 , srv1238458761:85 , srv1316756146:85 , srv1410174819:85 , srv1417345652:85 , srv1465452434:85 , srv1467441778:85 , srv148561286:85 , srv1502924087:85 , srv151066670:85 , srv1536020236:85 , srv1536762713:85 , srv1556507361:85 , srv1566881129:85 , srv1567381724:85 , srv1642939663:85 , srv1660995637:85 , srv1662818069:85 , srv1677231915:85 , srv1684632003:85 , srv1691854898:85 , srv1727500160:85 , srv1827974742:85 , srv1888148249:85 , srv1892675751:85 , srv1918913273:85 , srv2031688399:85 , srv2113366633:85 , srv2130760530:85 , srv221640249:85 , srv238145400:85 , srv254885644:85 , srv320919189:85 , srv384670142:85 , srv404937124:85 , srv455830398:85 , srv496619863:85 , srv562961356:85 , srv56512202:85 , srv607837081:85 , srv67611046:85 , srv756400963:85 , srv91941334:85 , srv1167692866:86 , srv1168181125:86 , srv1226322429:86 , srv142132921:86 , srv142200861:86 , srv1641429558:86 , srv1697694710:86 , srv2127034933:86 , srv223272156:86 , srv303537177:86 , srv563662290:86 , srv75645107:86 , srv1025831096:87 , srv2138150939:87 , srv479063391:612 } 2023-09-01 12:58:52,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:52,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv177487662=47, srv1626630171=34, srv1025831096=1, srv563662290=85, srv1641429558=36, srv151066670=25, srv1556507361=31, srv1502924087=23, srv23296469=66, srv303537177=72, srv1238458761=11, srv1691854898=42, srv2127034933=60, srv1124456074=4, srv1891071020=52, srv1417345652=17, srv309582065=74, srv142200861=19, srv468237166=81, srv1333079644=15, srv8402981=96, srv2130760530=61, srv1410174819=16, srv320919189=75, srv719246504=90, srv303428026=71, srv1969931799=56, srv404937124=78, srv436685318=79, srv1554886195=30, srv1245360656=12, srv1641359002=35, srv1660995637=38, srv2006088135=57, srv1789274169=48, srv244506920=69, srv1827974742=49, srv1892675751=53, srv1316756146=14, srv1109273719=3, srv1168181125=8, srv1099812634=2, srv75645107=93, srv240565309=68, srv712596408=89, srv1544251915=29, srv1888148249=51, srv347633784=76, srv1467441778=21, srv2031688399=58, srv1536020236=27, srv1677231915=40, srv1684632003=41, srv1953418212=55, srv223272156=65, srv254885644=70, srv148561286=22, srv959747368=99, srv765663733=94, srv112699978=5, srv1510149608=24, srv455830398=80, srv1727500160=45, srv1728157620=46, srv1226322429=9, srv1289054515=13, srv1167692866=7, srv936429796=98, srv1465452434=20, srv1697694710=43, srv2138150939=63, srv91941334=97, srv30473677=73, srv1642939663=37, srv562961356=84, srv1132490800=6, srv142132921=18, srv56512202=86, srv743654765=91, srv756400963=92, srv496619863=83, srv221640249=64, srv1015233269=0, srv1229931360=10, srv1703896406=44, srv1566881129=32, srv607837081=87, srv238145400=67, srv1918913273=54, srv1567381724=33, srv152336277=26, srv384670142=77, srv67611046=88, srv1832764600=50, srv1536762713=28, srv1662818069=39, srv2113366633=59, srv213743016=62, srv479063391=82, srv801993495=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:52,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-01 12:58:52,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-01 12:58:52,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-01 12:58:52,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-01 12:58:52,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-01 12:58:52,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-01 12:58:52,492 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,492 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-01 12:58:52,492 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=1.2353687253438515E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-01 12:58:52,674 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 259 ms to try 5000 different iterations. Found a solution that moves 34 regions; Going from a computed imbalance of 1.2353687253438515E-4 to a new imbalance of 7.6258305887507095E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,675 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1025831096:83 , srv1167692866:83 , srv1168181125:83 , srv1410174819:83 , srv1502924087:83 , srv1536020236:83 , srv1641429558:83 , srv1703896406:83 , srv2127034933:83 , srv2138150939:83 , srv223272156:83 , srv455830398:83 , srv75645107:83 , srv1015233269:84 , srv1099812634:84 , srv1124456074:84 , srv112699978:84 , srv1226322429:84 , srv1238458761:84 , srv1245360656:84 , srv1289054515:84 , srv1333079644:84 , srv1417345652:84 , srv142132921:84 , srv142200861:84 , srv1465452434:84 , srv1467441778:84 , srv148561286:84 , srv1510149608:84 , srv151066670:84 , srv1536762713:84 , srv1544251915:84 , srv1556507361:84 , srv1566881129:84 , srv1567381724:84 , srv1626630171:84 , srv1641359002:84 , srv1642939663:84 , srv1660995637:84 , srv1662818069:84 , srv1677231915:84 , srv1684632003:84 , srv1697694710:84 , srv1727500160:84 , srv177487662:84 , srv1789274169:84 , srv1827974742:84 , srv1888148249:84 , srv1891071020:84 , srv1892675751:84 , srv1918913273:84 , srv1953418212:84 , srv1969931799:84 , srv2006088135:84 , srv2031688399:84 , srv2113366633:84 , srv2130760530:84 , srv213743016:84 , srv221640249:84 , srv23296469:84 , srv238145400:84 , srv240565309:84 , srv244506920:84 , srv254885644:84 , srv303428026:84 , srv303537177:84 , srv320919189:84 , srv347633784:84 , srv384670142:84 , srv404937124:84 , srv436685318:84 , srv468237166:84 , srv496619863:84 , srv562961356:84 , srv563662290:84 , srv56512202:84 , srv607837081:84 , srv67611046:84 , srv719246504:84 , srv743654765:84 , srv756400963:84 , srv765663733:84 , srv801993495:84 , srv8402981:84 , srv91941334:84 , srv959747368:84 , srv1109273719:85 , srv1132490800:85 , srv1229931360:85 , srv1316756146:85 , srv152336277:85 , srv1554886195:85 , srv1691854898:85 , srv1728157620:85 , srv1832764600:85 , srv30473677:85 , srv309582065:85 , srv712596408:85 , srv936429796:85 , srv479063391:684 } 2023-09-01 12:58:52,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:52,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv177487662=47, srv1626630171=34, srv1025831096=1, srv563662290=85, srv1641429558=36, srv151066670=25, srv1556507361=31, srv1502924087=23, srv23296469=66, srv303537177=72, srv1238458761=11, srv1691854898=42, srv2127034933=60, srv1124456074=4, srv1891071020=52, srv1417345652=17, srv309582065=74, srv142200861=19, srv468237166=81, srv1333079644=15, srv8402981=96, srv2130760530=61, srv1410174819=16, srv320919189=75, srv719246504=90, srv303428026=71, srv1969931799=56, srv404937124=78, srv436685318=79, srv1554886195=30, srv1245360656=12, srv1641359002=35, srv1660995637=38, srv2006088135=57, srv1789274169=48, srv244506920=69, srv1827974742=49, srv1892675751=53, srv1316756146=14, srv1109273719=3, srv1168181125=8, srv1099812634=2, srv75645107=93, srv240565309=68, srv712596408=89, srv1544251915=29, srv1888148249=51, srv347633784=76, srv1467441778=21, srv2031688399=58, srv1536020236=27, srv1677231915=40, srv1684632003=41, srv1953418212=55, srv223272156=65, srv254885644=70, srv148561286=22, srv959747368=99, srv765663733=94, srv112699978=5, srv1510149608=24, srv455830398=80, srv1727500160=45, srv1728157620=46, srv1226322429=9, srv1289054515=13, srv1167692866=7, srv936429796=98, srv1465452434=20, srv1697694710=43, srv2138150939=63, srv91941334=97, srv30473677=73, srv1642939663=37, srv562961356=84, srv1132490800=6, srv142132921=18, srv56512202=86, srv743654765=91, srv756400963=92, srv496619863=83, srv221640249=64, srv1015233269=0, srv1229931360=10, srv1703896406=44, srv1566881129=32, srv607837081=87, srv238145400=67, srv1918913273=54, srv1567381724=33, srv152336277=26, srv384670142=77, srv67611046=88, srv1832764600=50, srv1536762713=28, srv1662818069=39, srv2113366633=59, srv213743016=62, srv479063391=82, srv801993495=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:52,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:52,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:52,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:52,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:52,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:52,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:52,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:52,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:52,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:52,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:52,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:52,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:52,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-01 12:58:52,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-01 12:58:52,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-01 12:58:52,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-01 12:58:52,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-01 12:58:52,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-01 12:58:52,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-01 12:58:52,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-01 12:58:52,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-01 12:58:52,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-01 12:58:52,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-01 12:58:52,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-01 12:58:52,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-01 12:58:52,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-01 12:58:52,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-01 12:58:52,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-01 12:58:52,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-01 12:58:52,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-01 12:58:52,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-01 12:58:52,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-01 12:58:52,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-01 12:58:52,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-01 12:58:52,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-01 12:58:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-01 12:58:52,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-01 12:58:52,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-01 12:58:52,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-01 12:58:52,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-01 12:58:52,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-01 12:58:52,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-01 12:58:52,810 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,811 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1025831096:83 , srv1167692866:83 , srv1168181125:83 , srv1410174819:83 , srv1502924087:83 , srv1536020236:83 , srv1641429558:83 , srv1703896406:83 , srv2127034933:83 , srv2138150939:83 , srv223272156:83 , srv455830398:83 , srv75645107:83 , srv1015233269:84 , srv1099812634:84 , srv1124456074:84 , srv112699978:84 , srv1226322429:84 , srv1238458761:84 , srv1245360656:84 , srv1289054515:84 , srv1333079644:84 , srv1417345652:84 , srv142132921:84 , srv142200861:84 , srv1465452434:84 , srv1467441778:84 , srv148561286:84 , srv1510149608:84 , srv151066670:84 , srv1536762713:84 , srv1544251915:84 , srv1556507361:84 , srv1566881129:84 , srv1567381724:84 , srv1626630171:84 , srv1641359002:84 , srv1642939663:84 , srv1660995637:84 , srv1662818069:84 , srv1677231915:84 , srv1684632003:84 , srv1697694710:84 , srv1727500160:84 , srv177487662:84 , srv1789274169:84 , srv1827974742:84 , srv1888148249:84 , srv1891071020:84 , srv1892675751:84 , srv1918913273:84 , srv1953418212:84 , srv1969931799:84 , srv2006088135:84 , srv2031688399:84 , srv2113366633:84 , srv2130760530:84 , srv213743016:84 , srv221640249:84 , srv23296469:84 , srv238145400:84 , srv240565309:84 , srv244506920:84 , srv254885644:84 , srv303428026:84 , srv303537177:84 , srv320919189:84 , srv347633784:84 , srv384670142:84 , srv404937124:84 , srv436685318:84 , srv468237166:84 , srv496619863:84 , srv562961356:84 , srv563662290:84 , srv56512202:84 , srv607837081:84 , srv67611046:84 , srv719246504:84 , srv743654765:84 , srv756400963:84 , srv765663733:84 , srv801993495:84 , srv8402981:84 , srv91941334:84 , srv959747368:84 , srv1109273719:85 , srv1132490800:85 , srv1229931360:85 , srv1316756146:85 , srv152336277:85 , srv1554886195:85 , srv1691854898:85 , srv1728157620:85 , srv1832764600:85 , srv30473677:85 , srv309582065:85 , srv712596408:85 , srv936429796:85 , srv479063391:684 } 2023-09-01 12:58:52,844 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=276 (was 257) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=5513 (was 6038) 2023-09-01 12:58:52,853 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=276, ProcessCount=166, AvailableMemoryMB=5512 2023-09-01 12:58:52,854 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-01 12:58:52,854 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-01 12:58:52,864 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1111257876:84 , srv1243270395:84 , srv1250798629:84 , srv1392175531:84 , srv1443061864:84 , srv1484558453:84 , srv1491556561:84 , srv153775885:84 , srv156478158:84 , srv1648804152:84 , srv1828367563:84 , srv1848715145:84 , srv1852203746:84 , srv1967607195:84 , srv1984566573:84 , srv2099738705:84 , srv253157356:84 , srv269203666:84 , srv271197810:84 , srv359238057:84 , srv363259446:84 , srv380070905:84 , srv498615254:84 , srv527706315:84 , srv575973397:84 , srv615214249:84 , srv633534617:84 , srv640469324:84 , srv844201206:84 , srv1958470995:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-01 12:58:52,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 12:58:52,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv575973397=25, srv844201206=29, srv1484558453=5, srv633534617=27, srv615214249=26, srv1491556561=6, srv2099738705=16, srv269203666=18, srv1852203746=12, srv1967607195=14, srv640469324=28, srv253157356=17, srv1984566573=15, srv1243270395=1, srv498615254=23, srv359238057=20, srv1958470995=13, srv1443061864=4, srv527706315=24, srv363259446=21, srv1111257876=0, srv1848715145=11, srv1648804152=9, srv271197810=19, srv1250798629=2, srv153775885=7, srv1828367563=10, srv1392175531=3, srv380070905=22, srv156478158=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 12:58:52,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 12:58:52,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 12:58:52,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 12:58:52,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 12:58:52,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-01 12:58:52,896 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1231637200631752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 12:58:52,896 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912581128124534, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1231637200631752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-01 13:00:52,881 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9333136 different iterations. Found a solution that moves 1842 regions; Going from a computed imbalance of 0.9912581128124534 to a new imbalance of 4.2993198912066E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 13:00:52,888 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1111257876:90 , srv1243270395:90 , srv1250798629:90 , srv1392175531:90 , srv1443061864:90 , srv1484558453:90 , srv1491556561:90 , srv153775885:90 , srv156478158:90 , srv1648804152:90 , srv1828367563:90 , srv1848715145:90 , srv1852203746:90 , srv1958470995:90 , srv1967607195:90 , srv1984566573:90 , srv2099738705:90 , srv253157356:90 , srv269203666:90 , srv271197810:90 , srv359238057:90 , srv363259446:90 , srv380070905:90 , srv498615254:90 , srv527706315:90 , srv575973397:90 , srv615214249:90 , srv633534617:90 , srv640469324:90 , srv844201206:90 } 2023-09-01 13:00:52,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-01 13:00:52,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv575973397=25, srv844201206=29, srv1484558453=5, srv633534617=27, srv615214249=26, srv1491556561=6, srv2099738705=16, srv269203666=18, srv1852203746=12, srv1967607195=14, srv640469324=28, srv253157356=17, srv1984566573=15, srv1243270395=1, srv498615254=23, srv359238057=20, srv1958470995=13, srv1443061864=4, srv527706315=24, srv363259446=21, srv1111257876=0, srv1848715145=11, srv1648804152=9, srv271197810=19, srv1250798629=2, srv153775885=7, srv1828367563=10, srv1392175531=3, srv380070905=22, srv156478158=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-01 13:00:52,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-01 13:00:52,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-01 13:00:52,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-01 13:00:52,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-01 13:00:52,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-01 13:00:52,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-01 13:00:52,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-01 13:00:52,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-01 13:00:52,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-01 13:00:52,907 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-01 13:00:52,907 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1111257876:90 , srv1243270395:90 , srv1250798629:90 , srv1392175531:90 , srv1443061864:90 , srv1484558453:90 , srv1491556561:90 , srv153775885:90 , srv156478158:90 , srv1648804152:90 , srv1828367563:90 , srv1848715145:90 , srv1852203746:90 , srv1958470995:90 , srv1967607195:90 , srv1984566573:90 , srv2099738705:90 , srv253157356:90 , srv269203666:90 , srv271197810:90 , srv359238057:90 , srv363259446:90 , srv380070905:90 , srv498615254:90 , srv527706315:90 , srv575973397:90 , srv615214249:90 , srv633534617:90 , srv640469324:90 , srv844201206:90 } 2023-09-01 13:00:52,918 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 276), ProcessCount=166 (was 166), AvailableMemoryMB=5679 (was 5512) - AvailableMemoryMB LEAK? -