2023-10-12 17:53:24,857 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-12 17:53:25,138 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-12 17:53:25,244 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-12 17:53:25,317 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-12 17:53:25,325 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-12 17:53:25,338 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=282, ProcessCount=169, AvailableMemoryMB=8111 2023-10-12 17:53:25,340 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-12 17:53:25,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-12 17:53:25,466 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv10124521:84 , srv1024850064:84 , srv1029313582:84 , srv1052995592:84 , srv1073338418:84 , srv1102346402:84 , srv110242320:84 , srv1116470649:84 , srv1129182588:84 , srv1214549685:84 , srv126688335:84 , srv1326954184:84 , srv1347054467:84 , srv1386606078:84 , srv1410092188:84 , srv1428153410:84 , srv1430202888:84 , srv1454271184:84 , srv1480831320:84 , srv1501606583:84 , srv1530241473:84 , srv1588390739:84 , srv1664918486:84 , srv1677945260:84 , srv1703478310:84 , srv1721665031:84 , srv1724258975:84 , srv1734162711:84 , srv173510674:84 , srv1741799589:84 , srv1779933954:84 , srv1817870282:84 , srv1820263339:84 , srv1826132312:84 , srv1838760910:84 , srv1853825190:84 , srv1863724530:84 , srv1865010520:84 , srv1924410453:84 , srv196443899:84 , srv1983205680:84 , srv1990063099:84 , srv1992995726:84 , srv2018256197:84 , srv2025689764:84 , srv2071966349:84 , srv2076738086:84 , srv2081116708:84 , srv2084953511:84 , srv2108186007:84 , srv2126515641:84 , srv2130390307:84 , srv2131611840:84 , srv2132492371:84 , srv2139614921:84 , srv216122008:84 , srv224730345:84 , srv235925591:84 , srv238454853:84 , srv243219588:84 , srv272978315:84 , srv288124863:84 , srv303268363:84 , srv317878425:84 , srv329013371:84 , srv335105383:84 , srv355018329:84 , srv358641080:84 , srv398026501:84 , srv420928099:84 , srv453875361:84 , srv466139459:84 , srv480780100:84 , srv510713535:84 , srv53623022:84 , srv561063056:84 , srv566848657:84 , srv585807344:84 , srv593650148:84 , srv639236246:84 , srv747564153:84 , srv751345217:84 , srv782192279:84 , srv793255874:84 , srv802094742:84 , srv854149975:84 , srv863249779:84 , srv901522389:84 , srv917720549:84 , srv922302188:84 , srv950009804:84 , srv965682052:84 , srv977432585:84 , srv979418066:84 , srv985084349:84 , srv986059031:84 , srv987763540:84 , srv990817381:84 , srv992393639:84 , srv227820616:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-12 17:53:25,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:25,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv466139459=72, srv453875361=71, srv288124863=62, srv1741799589=29, srv1588390739=21, srv985084349=95, srv1326954184=11, srv1129182588=8, srv216122008=55, srv2081116708=47, srv1677945260=23, srv2025689764=44, srv1214549685=9, srv990817381=98, srv1865010520=37, srv224730345=56, srv420928099=70, srv1863724530=36, srv1924410453=38, srv238454853=59, srv2071966349=45, srv1721665031=25, srv10124521=0, srv639236246=80, srv585807344=78, srv1102346402=5, srv854149975=86, srv1347054467=12, srv2108186007=49, srv2084953511=48, srv1052995592=3, srv53623022=75, srv110242320=6, srv1817870282=31, srv2018256197=43, srv1386606078=13, srv1853825190=35, srv561063056=76, srv1703478310=24, srv1983205680=40, srv1992995726=42, srv303268363=63, srv510713535=74, srv965682052=92, srv1990063099=41, srv917720549=89, srv1530241473=20, srv1826132312=33, srv1480831320=18, srv1430202888=16, srv1501606583=19, srv1664918486=22, srv398026501=69, srv355018329=67, srv782192279=83, srv358641080=68, srv243219588=60, srv2076738086=46, srv802094742=85, srv986059031=96, srv272978315=61, srv1779933954=30, srv2126515641=50, srv1454271184=17, srv335105383=66, srv793255874=84, srv977432585=93, srv747564153=81, srv1734162711=27, srv566848657=77, srv1724258975=26, srv196443899=39, srv1838760910=34, srv987763540=97, srv1029313582=2, srv126688335=10, srv1024850064=1, srv1073338418=4, srv1116470649=7, srv1428153410=15, srv329013371=65, srv901522389=88, srv950009804=91, srv235925591=58, srv922302188=90, srv979418066=94, srv1820263339=32, srv2130390307=51, srv2131611840=52, srv317878425=64, srv173510674=28, srv2132492371=53, srv593650148=79, srv480780100=73, srv227820616=57, srv992393639=99, srv2139614921=54, srv1410092188=14, srv863249779=87, srv751345217=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:25,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:25,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:25,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:25,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 17:53:25,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 17:53:25,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 17:53:25,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 17:53:25,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 17:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 17:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 17:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 17:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 17:53:25,812 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:25,813 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 17:53:25,813 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 17:53:26,268 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 671 ms to try 5000 different iterations. Found a solution that moves 4697 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.180691240610694. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949014124775387, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05505430438191396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5218888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949014124775387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24433333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:26,291 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2018256197:80 , srv585807344:81 , srv1428153410:82 , srv1820263339:82 , srv922302188:82 , srv1983205680:83 , srv2025689764:83 , srv224730345:83 , srv355018329:83 , srv950009804:83 , srv1073338418:84 , srv1214549685:84 , srv1924410453:84 , srv2131611840:84 , srv235925591:84 , srv335105383:84 , srv782192279:84 , srv986059031:84 , srv990817381:84 , srv1326954184:85 , srv1430202888:85 , srv1480831320:85 , srv1501606583:85 , srv1677945260:85 , srv1734162711:85 , srv1741799589:85 , srv1817870282:85 , srv196443899:85 , srv1992995726:85 , srv2081116708:85 , srv2132492371:85 , srv303268363:85 , srv329013371:85 , srv420928099:85 , srv453875361:85 , srv466139459:85 , srv53623022:85 , srv566848657:85 , srv863249779:85 , srv1530241473:86 , srv1588390739:86 , srv1664918486:86 , srv1703478310:86 , srv2076738086:86 , srv2139614921:86 , srv216122008:86 , srv243219588:86 , srv317878425:86 , srv593650148:86 , srv751345217:86 , srv793255874:86 , srv802094742:86 , srv10124521:87 , srv1052995592:87 , srv1386606078:87 , srv1724258975:87 , srv1779933954:87 , srv1853825190:87 , srv1863724530:87 , srv1865010520:87 , srv2071966349:87 , srv2084953511:87 , srv2108186007:87 , srv2126515641:87 , srv358641080:87 , srv747564153:87 , srv992393639:87 , srv1129182588:88 , srv126688335:88 , srv1454271184:88 , srv1990063099:88 , srv2130390307:88 , srv272978315:88 , srv510713535:88 , srv561063056:88 , srv639236246:88 , srv917720549:88 , srv979418066:88 , srv1410092188:89 , srv1721665031:89 , srv288124863:89 , srv398026501:89 , srv480780100:89 , srv901522389:89 , srv965682052:89 , srv985084349:89 , srv987763540:89 , srv1024850064:90 , srv1029313582:90 , srv1102346402:90 , srv1347054467:90 , srv173510674:90 , srv854149975:90 , srv1116470649:91 , srv1826132312:91 , srv238454853:91 , srv110242320:92 , srv1838760910:92 , srv977432585:92 , srv227820616:441 } 2023-10-12 17:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:26,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv466139459=72, srv453875361=71, srv288124863=62, srv1741799589=29, srv1588390739=21, srv985084349=95, srv1326954184=11, srv1129182588=8, srv216122008=55, srv2081116708=47, srv1677945260=23, srv2025689764=44, srv1214549685=9, srv990817381=98, srv1865010520=37, srv224730345=56, srv420928099=70, srv1863724530=36, srv1924410453=38, srv238454853=59, srv2071966349=45, srv1721665031=25, srv10124521=0, srv639236246=80, srv585807344=78, srv1102346402=5, srv854149975=86, srv1347054467=12, srv2108186007=49, srv2084953511=48, srv1052995592=3, srv53623022=75, srv110242320=6, srv1817870282=31, srv2018256197=43, srv1386606078=13, srv1853825190=35, srv561063056=76, srv1703478310=24, srv1983205680=40, srv1992995726=42, srv303268363=63, srv510713535=74, srv965682052=92, srv1990063099=41, srv917720549=89, srv1530241473=20, srv1826132312=33, srv1480831320=18, srv1430202888=16, srv1501606583=19, srv1664918486=22, srv398026501=69, srv355018329=67, srv782192279=83, srv358641080=68, srv243219588=60, srv2076738086=46, srv802094742=85, srv986059031=96, srv272978315=61, srv1779933954=30, srv2126515641=50, srv1454271184=17, srv335105383=66, srv793255874=84, srv977432585=93, srv747564153=81, srv1734162711=27, srv566848657=77, srv1724258975=26, srv196443899=39, srv1838760910=34, srv987763540=97, srv1029313582=2, srv126688335=10, srv1024850064=1, srv1073338418=4, srv1116470649=7, srv1428153410=15, srv329013371=65, srv901522389=88, srv950009804=91, srv235925591=58, srv922302188=90, srv979418066=94, srv1820263339=32, srv2130390307=51, srv2131611840=52, srv317878425=64, srv173510674=28, srv2132492371=53, srv593650148=79, srv480780100=73, srv227820616=57, srv992393639=99, srv2139614921=54, srv1410092188=14, srv863249779=87, srv751345217=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:26,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:26,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:26,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:26,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:26,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:26,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:26,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:26,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:26,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:26,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:26,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 17:53:26,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 17:53:26,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 17:53:26,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 17:53:26,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 17:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 17:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 17:53:26,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 17:53:26,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 17:53:26,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 17:53:26,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 17:53:26,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 17:53:26,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 17:53:26,560 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949014124775387, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05505430438191396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949014124775387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24433333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:26,561 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 17:53:26,561 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18065835151374127, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949014124775387, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05505430438191396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949014124775387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24433333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 17:53:26,868 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 499 ms to try 5000 different iterations. Found a solution that moves 2669 regions; Going from a computed imbalance of 0.18065835151374127 to a new imbalance of 0.01450490804069294. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020938213370522325); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.025767674787862873, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29655555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020938213370522325); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:26,880 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1347054467:80 , srv1838760910:80 , srv1024850064:81 , srv1102346402:81 , srv901522389:81 , srv977432585:81 , srv1116470649:82 , srv1129182588:82 , srv1454271184:82 , srv173510674:82 , srv561063056:82 , srv854149975:82 , srv987763540:82 , srv126688335:83 , srv1410092188:83 , srv1826132312:83 , srv238454853:83 , srv398026501:83 , srv639236246:83 , srv992393639:83 , srv10124521:84 , srv1029313582:84 , srv110242320:84 , srv1386606078:84 , srv1721665031:84 , srv1724258975:84 , srv1853825190:84 , srv1990063099:84 , srv2139614921:84 , srv272978315:84 , srv288124863:84 , srv358641080:84 , srv979418066:84 , srv1779933954:85 , srv1992995726:85 , srv2132492371:85 , srv480780100:85 , srv510713535:85 , srv566848657:85 , srv593650148:85 , srv751345217:85 , srv863249779:85 , srv985084349:85 , srv1052995592:86 , srv1326954184:86 , srv1430202888:86 , srv1588390739:86 , srv1677945260:86 , srv2071966349:86 , srv2081116708:86 , srv2084953511:86 , srv2126515641:86 , srv2130390307:86 , srv243219588:86 , srv303268363:86 , srv317878425:86 , srv453875361:86 , srv747564153:86 , srv917720549:86 , srv965682052:86 , srv1530241473:87 , srv1734162711:87 , srv1741799589:87 , srv1863724530:87 , srv1865010520:87 , srv2076738086:87 , srv335105383:87 , srv355018329:87 , srv53623022:87 , srv782192279:87 , srv802094742:87 , srv950009804:87 , srv986059031:87 , srv990817381:87 , srv1501606583:88 , srv1664918486:88 , srv1817870282:88 , srv1924410453:88 , srv196443899:88 , srv2025689764:88 , srv2108186007:88 , srv2131611840:88 , srv216122008:88 , srv329013371:88 , srv420928099:88 , srv466139459:88 , srv1073338418:89 , srv1214549685:89 , srv1428153410:89 , srv1480831320:89 , srv1820263339:89 , srv1983205680:89 , srv235925591:89 , srv793255874:89 , srv1703478310:90 , srv585807344:90 , srv922302188:90 , srv2018256197:91 , srv224730345:91 , srv227820616:519 } 2023-10-12 17:53:26,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:26,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv466139459=72, srv453875361=71, srv288124863=62, srv1741799589=29, srv1588390739=21, srv985084349=95, srv1326954184=11, srv1129182588=8, srv216122008=55, srv2081116708=47, srv1677945260=23, srv2025689764=44, srv1214549685=9, srv990817381=98, srv1865010520=37, srv224730345=56, srv420928099=70, srv1863724530=36, srv1924410453=38, srv238454853=59, srv2071966349=45, srv1721665031=25, srv10124521=0, srv639236246=80, srv585807344=78, srv1102346402=5, srv854149975=86, srv1347054467=12, srv2108186007=49, srv2084953511=48, srv1052995592=3, srv53623022=75, srv110242320=6, srv1817870282=31, srv2018256197=43, srv1386606078=13, srv1853825190=35, srv561063056=76, srv1703478310=24, srv1983205680=40, srv1992995726=42, srv303268363=63, srv510713535=74, srv965682052=92, srv1990063099=41, srv917720549=89, srv1530241473=20, srv1826132312=33, srv1480831320=18, srv1430202888=16, srv1501606583=19, srv1664918486=22, srv398026501=69, srv355018329=67, srv782192279=83, srv358641080=68, srv243219588=60, srv2076738086=46, srv802094742=85, srv986059031=96, srv272978315=61, srv1779933954=30, srv2126515641=50, srv1454271184=17, srv335105383=66, srv793255874=84, srv977432585=93, srv747564153=81, srv1734162711=27, srv566848657=77, srv1724258975=26, srv196443899=39, srv1838760910=34, srv987763540=97, srv1029313582=2, srv126688335=10, srv1024850064=1, srv1073338418=4, srv1116470649=7, srv1428153410=15, srv329013371=65, srv901522389=88, srv950009804=91, srv235925591=58, srv922302188=90, srv979418066=94, srv1820263339=32, srv2130390307=51, srv2131611840=52, srv317878425=64, srv173510674=28, srv2132492371=53, srv593650148=79, srv480780100=73, srv227820616=57, srv992393639=99, srv2139614921=54, srv1410092188=14, srv863249779=87, srv751345217=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:26,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:26,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:26,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 17:53:26,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 17:53:26,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 17:53:26,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 17:53:26,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 17:53:26,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 17:53:26,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 17:53:26,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 17:53:26,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 17:53:26,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 17:53:27,067 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020938213370522325); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.025767674787862873, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020938213370522325); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:27,067 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 17:53:27,067 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014486219303250546, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020938213370522325); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.025767674787862873, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020938213370522325); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 17:53:27,463 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 523 ms to try 5000 different iterations. Found a solution that moves 995 regions; Going from a computed imbalance of 0.014486219303250546 to a new imbalance of 6.900941165070059E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058218983640361864); MoveCostFunction : (multiplier=7.0, imbalance=0.11055555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:27,465 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1703478310:81 , srv110242320:82 , srv1826132312:82 , srv2108186007:82 , srv480780100:82 , srv1480831320:83 , srv1530241473:83 , srv1865010520:83 , srv224730345:83 , srv238454853:83 , srv793255874:83 , srv917720549:83 , srv965682052:83 , srv1029313582:84 , srv1073338418:84 , srv1214549685:84 , srv1501606583:84 , srv1664918486:84 , srv1721665031:84 , srv1863724530:84 , srv2071966349:84 , srv2084953511:84 , srv2126515641:84 , srv2130390307:84 , srv235925591:84 , srv466139459:84 , srv510713535:84 , srv747564153:84 , srv802094742:84 , srv977432585:84 , srv985084349:84 , srv1052995592:85 , srv1116470649:85 , srv1410092188:85 , srv1588390739:85 , srv173510674:85 , srv1779933954:85 , srv1838760910:85 , srv196443899:85 , srv1983205680:85 , srv2076738086:85 , srv216122008:85 , srv243219588:85 , srv288124863:85 , srv317878425:85 , srv329013371:85 , srv398026501:85 , srv420928099:85 , srv782192279:85 , srv922302188:85 , srv979418066:85 , srv10124521:86 , srv1024850064:86 , srv1102346402:86 , srv126688335:86 , srv1326954184:86 , srv1347054467:86 , srv1386606078:86 , srv1428153410:86 , srv1430202888:86 , srv1677945260:86 , srv1724258975:86 , srv1734162711:86 , srv1741799589:86 , srv1817870282:86 , srv1853825190:86 , srv1924410453:86 , srv1990063099:86 , srv2018256197:86 , srv2025689764:86 , srv272978315:86 , srv303268363:86 , srv335105383:86 , srv453875361:86 , srv53623022:86 , srv585807344:86 , srv593650148:86 , srv751345217:86 , srv854149975:86 , srv986059031:86 , srv987763540:86 , srv1820263339:87 , srv2081116708:87 , srv2131611840:87 , srv2132492371:87 , srv2139614921:87 , srv355018329:87 , srv358641080:87 , srv566848657:87 , srv639236246:87 , srv992393639:87 , srv1129182588:88 , srv1454271184:88 , srv1992995726:88 , srv561063056:88 , srv863249779:88 , srv901522389:88 , srv950009804:88 , srv990817381:88 , srv227820616:561 } 2023-10-12 17:53:27,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:27,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv466139459=72, srv453875361=71, srv288124863=62, srv1741799589=29, srv1588390739=21, srv985084349=95, srv1326954184=11, srv1129182588=8, srv216122008=55, srv2081116708=47, srv1677945260=23, srv2025689764=44, srv1214549685=9, srv990817381=98, srv1865010520=37, srv224730345=56, srv420928099=70, srv1863724530=36, srv1924410453=38, srv238454853=59, srv2071966349=45, srv1721665031=25, srv10124521=0, srv639236246=80, srv585807344=78, srv1102346402=5, srv854149975=86, srv1347054467=12, srv2108186007=49, srv2084953511=48, srv1052995592=3, srv53623022=75, srv110242320=6, srv1817870282=31, srv2018256197=43, srv1386606078=13, srv1853825190=35, srv561063056=76, srv1703478310=24, srv1983205680=40, srv1992995726=42, srv303268363=63, srv510713535=74, srv965682052=92, srv1990063099=41, srv917720549=89, srv1530241473=20, srv1826132312=33, srv1480831320=18, srv1430202888=16, srv1501606583=19, srv1664918486=22, srv398026501=69, srv355018329=67, srv782192279=83, srv358641080=68, srv243219588=60, srv2076738086=46, srv802094742=85, srv986059031=96, srv272978315=61, srv1779933954=30, srv2126515641=50, srv1454271184=17, srv335105383=66, srv793255874=84, srv977432585=93, srv747564153=81, srv1734162711=27, srv566848657=77, srv1724258975=26, srv196443899=39, srv1838760910=34, srv987763540=97, srv1029313582=2, srv126688335=10, srv1024850064=1, srv1073338418=4, srv1116470649=7, srv1428153410=15, srv329013371=65, srv901522389=88, srv950009804=91, srv235925591=58, srv922302188=90, srv979418066=94, srv1820263339=32, srv2130390307=51, srv2131611840=52, srv317878425=64, srv173510674=28, srv2132492371=53, srv593650148=79, srv480780100=73, srv227820616=57, srv992393639=99, srv2139614921=54, srv1410092188=14, srv863249779=87, srv751345217=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:27,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:27,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:27,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:27,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:27,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:27,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 17:53:27,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 17:53:27,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 17:53:27,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 17:53:27,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 17:53:27,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 17:53:27,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 17:53:27,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 17:53:27,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 17:53:27,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 17:53:27,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 17:53:27,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 17:53:27,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058218983640361864); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:27,618 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 17:53:27,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.831269775953602E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058218983640361864); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 17:53:27,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 424 ms to try 5000 different iterations. Found a solution that moves 211 regions; Going from a computed imbalance of 6.831269775953602E-4 to a new imbalance of 2.5076598104153466E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.023444444444444445); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:27,938 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1454271184:83 , srv1817870282:83 , srv2131611840:83 , srv216122008:83 , srv990817381:83 , srv1029313582:84 , srv1052995592:84 , srv110242320:84 , srv1116470649:84 , srv1129182588:84 , srv1664918486:84 , srv1734162711:84 , srv1741799589:84 , srv1820263339:84 , srv1863724530:84 , srv1924410453:84 , srv196443899:84 , srv1990063099:84 , srv1992995726:84 , srv2076738086:84 , srv2081116708:84 , srv2126515641:84 , srv2130390307:84 , srv272978315:84 , srv288124863:84 , srv329013371:84 , srv358641080:84 , srv420928099:84 , srv53623022:84 , srv561063056:84 , srv639236246:84 , srv793255874:84 , srv854149975:84 , srv863249779:84 , srv901522389:84 , srv922302188:84 , srv950009804:84 , srv965682052:84 , srv985084349:84 , srv10124521:85 , srv1024850064:85 , srv1073338418:85 , srv1102346402:85 , srv1214549685:85 , srv126688335:85 , srv1326954184:85 , srv1386606078:85 , srv1410092188:85 , srv1430202888:85 , srv1480831320:85 , srv1501606583:85 , srv1588390739:85 , srv1677945260:85 , srv1721665031:85 , srv1724258975:85 , srv173510674:85 , srv1779933954:85 , srv1826132312:85 , srv1838760910:85 , srv1853825190:85 , srv1865010520:85 , srv1983205680:85 , srv2018256197:85 , srv2025689764:85 , srv2071966349:85 , srv2084953511:85 , srv2108186007:85 , srv2132492371:85 , srv2139614921:85 , srv224730345:85 , srv235925591:85 , srv243219588:85 , srv303268363:85 , srv317878425:85 , srv335105383:85 , srv355018329:85 , srv398026501:85 , srv453875361:85 , srv510713535:85 , srv566848657:85 , srv585807344:85 , srv593650148:85 , srv747564153:85 , srv751345217:85 , srv802094742:85 , srv917720549:85 , srv977432585:85 , srv979418066:85 , srv986059031:85 , srv987763540:85 , srv992393639:85 , srv1347054467:86 , srv1428153410:86 , srv1530241473:86 , srv1703478310:86 , srv238454853:86 , srv466139459:86 , srv480780100:86 , srv782192279:86 , srv227820616:621 } 2023-10-12 17:53:27,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:27,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv466139459=72, srv453875361=71, srv288124863=62, srv1741799589=29, srv1588390739=21, srv985084349=95, srv1326954184=11, srv1129182588=8, srv216122008=55, srv2081116708=47, srv1677945260=23, srv2025689764=44, srv1214549685=9, srv990817381=98, srv1865010520=37, srv224730345=56, srv420928099=70, srv1863724530=36, srv1924410453=38, srv238454853=59, srv2071966349=45, srv1721665031=25, srv10124521=0, srv639236246=80, srv585807344=78, srv1102346402=5, srv854149975=86, srv1347054467=12, srv2108186007=49, srv2084953511=48, srv1052995592=3, srv53623022=75, srv110242320=6, srv1817870282=31, srv2018256197=43, srv1386606078=13, srv1853825190=35, srv561063056=76, srv1703478310=24, srv1983205680=40, srv1992995726=42, srv303268363=63, srv510713535=74, srv965682052=92, srv1990063099=41, srv917720549=89, srv1530241473=20, srv1826132312=33, srv1480831320=18, srv1430202888=16, srv1501606583=19, srv1664918486=22, srv398026501=69, srv355018329=67, srv782192279=83, srv358641080=68, srv243219588=60, srv2076738086=46, srv802094742=85, srv986059031=96, srv272978315=61, srv1779933954=30, srv2126515641=50, srv1454271184=17, srv335105383=66, srv793255874=84, srv977432585=93, srv747564153=81, srv1734162711=27, srv566848657=77, srv1724258975=26, srv196443899=39, srv1838760910=34, srv987763540=97, srv1029313582=2, srv126688335=10, srv1024850064=1, srv1073338418=4, srv1116470649=7, srv1428153410=15, srv329013371=65, srv901522389=88, srv950009804=91, srv235925591=58, srv922302188=90, srv979418066=94, srv1820263339=32, srv2130390307=51, srv2131611840=52, srv317878425=64, srv173510674=28, srv2132492371=53, srv593650148=79, srv480780100=73, srv227820616=57, srv992393639=99, srv2139614921=54, srv1410092188=14, srv863249779=87, srv751345217=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:27,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:27,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:27,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 17:53:27,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 17:53:27,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 17:53:27,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 17:53:27,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 17:53:27,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 17:53:27,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 17:53:27,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 17:53:28,103 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:28,104 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 17:53:28,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.359914452590495E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 17:53:28,439 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations. Found a solution that moves 14 regions; Going from a computed imbalance of 2.359914452590495E-5 to a new imbalance of 6.496015767508937E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0015555555555555555); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:28,440 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1428153410:83 , srv1703478310:83 , srv238454853:83 , srv466139459:83 , srv10124521:84 , srv1024850064:84 , srv1029313582:84 , srv1052995592:84 , srv1073338418:84 , srv1102346402:84 , srv110242320:84 , srv1116470649:84 , srv1129182588:84 , srv1214549685:84 , srv126688335:84 , srv1326954184:84 , srv1347054467:84 , srv1386606078:84 , srv1410092188:84 , srv1430202888:84 , srv1480831320:84 , srv1501606583:84 , srv1530241473:84 , srv1588390739:84 , srv1664918486:84 , srv1677945260:84 , srv1721665031:84 , srv1724258975:84 , srv1734162711:84 , srv173510674:84 , srv1741799589:84 , srv1779933954:84 , srv1817870282:84 , srv1820263339:84 , srv1838760910:84 , srv1853825190:84 , srv1863724530:84 , srv1865010520:84 , srv1924410453:84 , srv196443899:84 , srv1983205680:84 , srv1990063099:84 , srv1992995726:84 , srv2018256197:84 , srv2025689764:84 , srv2071966349:84 , srv2076738086:84 , srv2081116708:84 , srv2084953511:84 , srv2108186007:84 , srv2130390307:84 , srv2131611840:84 , srv2132492371:84 , srv2139614921:84 , srv216122008:84 , srv224730345:84 , srv235925591:84 , srv243219588:84 , srv272978315:84 , srv288124863:84 , srv303268363:84 , srv317878425:84 , srv329013371:84 , srv335105383:84 , srv355018329:84 , srv358641080:84 , srv398026501:84 , srv420928099:84 , srv453875361:84 , srv480780100:84 , srv510713535:84 , srv53623022:84 , srv561063056:84 , srv566848657:84 , srv585807344:84 , srv593650148:84 , srv639236246:84 , srv747564153:84 , srv751345217:84 , srv782192279:84 , srv793255874:84 , srv802094742:84 , srv854149975:84 , srv863249779:84 , srv901522389:84 , srv917720549:84 , srv950009804:84 , srv965682052:84 , srv977432585:84 , srv979418066:84 , srv985084349:84 , srv986059031:84 , srv987763540:84 , srv990817381:84 , srv992393639:84 , srv1454271184:85 , srv1826132312:85 , srv2126515641:85 , srv922302188:85 , srv227820616:684 } 2023-10-12 17:53:28,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:28,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv466139459=72, srv453875361=71, srv288124863=62, srv1741799589=29, srv1588390739=21, srv985084349=95, srv1326954184=11, srv1129182588=8, srv216122008=55, srv2081116708=47, srv1677945260=23, srv2025689764=44, srv1214549685=9, srv990817381=98, srv1865010520=37, srv224730345=56, srv420928099=70, srv1863724530=36, srv1924410453=38, srv238454853=59, srv2071966349=45, srv1721665031=25, srv10124521=0, srv639236246=80, srv585807344=78, srv1102346402=5, srv854149975=86, srv1347054467=12, srv2108186007=49, srv2084953511=48, srv1052995592=3, srv53623022=75, srv110242320=6, srv1817870282=31, srv2018256197=43, srv1386606078=13, srv1853825190=35, srv561063056=76, srv1703478310=24, srv1983205680=40, srv1992995726=42, srv303268363=63, srv510713535=74, srv965682052=92, srv1990063099=41, srv917720549=89, srv1530241473=20, srv1826132312=33, srv1480831320=18, srv1430202888=16, srv1501606583=19, srv1664918486=22, srv398026501=69, srv355018329=67, srv782192279=83, srv358641080=68, srv243219588=60, srv2076738086=46, srv802094742=85, srv986059031=96, srv272978315=61, srv1779933954=30, srv2126515641=50, srv1454271184=17, srv335105383=66, srv793255874=84, srv977432585=93, srv747564153=81, srv1734162711=27, srv566848657=77, srv1724258975=26, srv196443899=39, srv1838760910=34, srv987763540=97, srv1029313582=2, srv126688335=10, srv1024850064=1, srv1073338418=4, srv1116470649=7, srv1428153410=15, srv329013371=65, srv901522389=88, srv950009804=91, srv235925591=58, srv922302188=90, srv979418066=94, srv1820263339=32, srv2130390307=51, srv2131611840=52, srv317878425=64, srv173510674=28, srv2132492371=53, srv593650148=79, srv480780100=73, srv227820616=57, srv992393639=99, srv2139614921=54, srv1410092188=14, srv863249779=87, srv751345217=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:28,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:28,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:28,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 17:53:28,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 17:53:28,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 17:53:28,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 17:53:28,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 17:53:28,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 17:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 17:53:28,637 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:28,638 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1428153410:83 , srv1703478310:83 , srv238454853:83 , srv466139459:83 , srv10124521:84 , srv1024850064:84 , srv1029313582:84 , srv1052995592:84 , srv1073338418:84 , srv1102346402:84 , srv110242320:84 , srv1116470649:84 , srv1129182588:84 , srv1214549685:84 , srv126688335:84 , srv1326954184:84 , srv1347054467:84 , srv1386606078:84 , srv1410092188:84 , srv1430202888:84 , srv1480831320:84 , srv1501606583:84 , srv1530241473:84 , srv1588390739:84 , srv1664918486:84 , srv1677945260:84 , srv1721665031:84 , srv1724258975:84 , srv1734162711:84 , srv173510674:84 , srv1741799589:84 , srv1779933954:84 , srv1817870282:84 , srv1820263339:84 , srv1838760910:84 , srv1853825190:84 , srv1863724530:84 , srv1865010520:84 , srv1924410453:84 , srv196443899:84 , srv1983205680:84 , srv1990063099:84 , srv1992995726:84 , srv2018256197:84 , srv2025689764:84 , srv2071966349:84 , srv2076738086:84 , srv2081116708:84 , srv2084953511:84 , srv2108186007:84 , srv2130390307:84 , srv2131611840:84 , srv2132492371:84 , srv2139614921:84 , srv216122008:84 , srv224730345:84 , srv235925591:84 , srv243219588:84 , srv272978315:84 , srv288124863:84 , srv303268363:84 , srv317878425:84 , srv329013371:84 , srv335105383:84 , srv355018329:84 , srv358641080:84 , srv398026501:84 , srv420928099:84 , srv453875361:84 , srv480780100:84 , srv510713535:84 , srv53623022:84 , srv561063056:84 , srv566848657:84 , srv585807344:84 , srv593650148:84 , srv639236246:84 , srv747564153:84 , srv751345217:84 , srv782192279:84 , srv793255874:84 , srv802094742:84 , srv854149975:84 , srv863249779:84 , srv901522389:84 , srv917720549:84 , srv950009804:84 , srv965682052:84 , srv977432585:84 , srv979418066:84 , srv985084349:84 , srv986059031:84 , srv987763540:84 , srv990817381:84 , srv992393639:84 , srv1454271184:85 , srv1826132312:85 , srv2126515641:85 , srv922302188:85 , srv227820616:684 } 2023-10-12 17:53:28,676 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=282 (was 282), ProcessCount=169 (was 169), AvailableMemoryMB=8050 (was 8111) 2023-10-12 17:53:28,687 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=282, ProcessCount=169, AvailableMemoryMB=8049 2023-10-12 17:53:28,688 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-12 17:53:28,688 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-12 17:53:28,708 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1178027600:84 , srv1222467124:84 , srv1241777397:84 , srv1272903911:84 , srv1353204957:84 , srv1446550025:84 , srv1612080115:84 , srv1693733103:84 , srv1909397089:84 , srv200528269:84 , srv201419714:84 , srv210024536:84 , srv259316624:84 , srv405768218:84 , srv405916049:84 , srv46779510:84 , srv502062790:84 , srv582965080:84 , srv63386768:84 , srv637064821:84 , srv666593982:84 , srv719971035:84 , srv778717493:84 , srv832088255:84 , srv870103092:84 , srv897796478:84 , srv911270645:84 , srv961720066:84 , srv998142962:84 , srv633267215:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-12 17:53:28,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:53:28,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv832088255=24, srv1353204957=4, srv210024536=11, srv1909397089=8, srv778717493=23, srv961720066=28, srv666593982=21, srv259316624=12, srv633267215=18, srv46779510=15, srv502062790=16, srv1693733103=7, srv63386768=19, srv1178027600=0, srv870103092=25, srv719971035=22, srv405768218=13, srv911270645=27, srv405916049=14, srv1446550025=5, srv897796478=26, srv998142962=29, srv1612080115=6, srv582965080=17, srv201419714=10, srv1241777397=2, srv1272903911=3, srv637064821=20, srv1222467124=1, srv200528269=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:53:28,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:53:28,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:53:28,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-12 17:53:28,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1537792301183616, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:53:28,758 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912677596597028, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1537792301183616, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-12 17:55:28,736 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5402703 different iterations. Found a solution that moves 1871 regions; Going from a computed imbalance of 0.9912677596597028 to a new imbalance of 4.367007337919408E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.692962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:55:28,743 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1178027600:90 , srv1222467124:90 , srv1241777397:90 , srv1272903911:90 , srv1353204957:90 , srv1446550025:90 , srv1612080115:90 , srv1693733103:90 , srv1909397089:90 , srv200528269:90 , srv201419714:90 , srv210024536:90 , srv259316624:90 , srv405768218:90 , srv405916049:90 , srv46779510:90 , srv502062790:90 , srv582965080:90 , srv633267215:90 , srv63386768:90 , srv637064821:90 , srv666593982:90 , srv719971035:90 , srv778717493:90 , srv832088255:90 , srv870103092:90 , srv897796478:90 , srv911270645:90 , srv961720066:90 , srv998142962:90 } 2023-10-12 17:55:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 17:55:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv832088255=24, srv1353204957=4, srv210024536=11, srv1909397089=8, srv778717493=23, srv961720066=28, srv666593982=21, srv259316624=12, srv633267215=18, srv46779510=15, srv502062790=16, srv1693733103=7, srv63386768=19, srv1178027600=0, srv870103092=25, srv719971035=22, srv405768218=13, srv911270645=27, srv405916049=14, srv1446550025=5, srv897796478=26, srv998142962=29, srv1612080115=6, srv582965080=17, srv201419714=10, srv1241777397=2, srv1272903911=3, srv637064821=20, srv1222467124=1, srv200528269=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 17:55:28,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 17:55:28,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 17:55:28,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 17:55:28,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-12 17:55:28,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 17:55:28,763 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1178027600:90 , srv1222467124:90 , srv1241777397:90 , srv1272903911:90 , srv1353204957:90 , srv1446550025:90 , srv1612080115:90 , srv1693733103:90 , srv1909397089:90 , srv200528269:90 , srv201419714:90 , srv210024536:90 , srv259316624:90 , srv405768218:90 , srv405916049:90 , srv46779510:90 , srv502062790:90 , srv582965080:90 , srv633267215:90 , srv63386768:90 , srv637064821:90 , srv666593982:90 , srv719971035:90 , srv778717493:90 , srv832088255:90 , srv870103092:90 , srv897796478:90 , srv911270645:90 , srv961720066:90 , srv998142962:90 } 2023-10-12 17:55:28,776 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 282) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7456 (was 8049)