2023-05-15 16:53:25,878 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-15 16:53:26,207 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-15 16:53:26,323 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-15 16:53:26,388 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 16:53:26,396 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-15 16:53:26,413 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=274, ProcessCount=173, AvailableMemoryMB=14666 2023-05-15 16:53:26,415 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 16:53:26,416 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-15 16:53:26,560 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010433963:84 , srv1023985039:84 , srv1082245196:84 , srv1086612597:84 , srv1127839345:84 , srv1168756977:84 , srv1195046023:84 , srv1203836290:84 , srv1229468118:84 , srv1261291499:84 , srv1280681075:84 , srv1285759486:84 , srv1331477267:84 , srv1367025230:84 , srv1390298633:84 , srv1426403133:84 , srv1459004596:84 , srv1475623699:84 , srv149586711:84 , srv1499368511:84 , srv1510325172:84 , srv1510458907:84 , srv1551047654:84 , srv1570879323:84 , srv1589842295:84 , srv1591762379:84 , srv159391221:84 , srv1600256608:84 , srv1609151571:84 , srv1612617850:84 , srv1627567214:84 , srv1672626812:84 , srv1680382957:84 , srv168414492:84 , srv1729176716:84 , srv1745496204:84 , srv17471883:84 , srv1761579789:84 , srv1770285521:84 , srv1773373249:84 , srv1798990410:84 , srv1812224273:84 , srv1832171415:84 , srv1858838724:84 , srv1925851644:84 , srv1952167075:84 , srv1957986348:84 , srv195863619:84 , srv195891643:84 , srv1997099605:84 , srv2010042268:84 , srv2017970305:84 , srv20368600:84 , srv2075292639:84 , srv208631344:84 , srv2100525507:84 , srv2125757423:84 , srv280736633:84 , srv281718860:84 , srv32320901:84 , srv33714291:84 , srv367272591:84 , srv40092058:84 , srv457347245:84 , srv476097709:84 , srv501963783:84 , srv513371534:84 , srv515788123:84 , srv524480463:84 , srv527266708:84 , srv557002828:84 , srv56019388:84 , srv564094276:84 , srv597007943:84 , srv619456630:84 , srv632138935:84 , srv635451662:84 , srv635845228:84 , srv64340107:84 , srv652864000:84 , srv667254083:84 , srv673867203:84 , srv686463410:84 , srv694581975:84 , srv703632214:84 , srv707163970:84 , srv738094660:84 , srv747221021:84 , srv757792109:84 , srv792074520:84 , srv801129063:84 , srv812818676:84 , srv813786702:84 , srv825854433:84 , srv929223649:84 , srv949499240:84 , srv957582320:84 , srv992747743:84 , srv992917975:84 , srv71802562:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-15 16:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:53:26,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2017970305=51, srv1426403133=15, srv280736633=57, srv1997099605=49, srv1127839345=4, srv168414492=33, srv1082245196=2, srv1591762379=25, srv1729176716=34, srv32320901=59, srv501963783=65, srv56019388=71, srv1957986348=46, srv1570879323=23, srv1229468118=8, srv33714291=60, srv1168756977=5, srv1672626812=31, srv1390298633=14, srv1023985039=1, srv20368600=52, srv792074520=90, srv801129063=91, srv825854433=94, srv557002828=70, srv71802562=86, srv1203836290=7, srv195891643=48, srv992917975=99, srv1609151571=28, srv281718860=58, srv1773373249=39, srv1510325172=20, srv1589842295=24, srv694581975=83, srv476097709=64, srv652864000=79, srv1285759486=11, srv1475623699=17, srv635845228=77, srv40092058=62, srv1798990410=40, srv1761579789=37, srv667254083=80, srv1858838724=43, srv2075292639=53, srv2125757423=56, srv17471883=36, srv1745496204=35, srv619456630=74, srv513371534=66, srv524480463=68, srv597007943=73, srv1812224273=41, srv1600256608=27, srv1459004596=16, srv992747743=98, srv635451662=76, srv1551047654=22, srv149586711=18, srv1627567214=30, srv738094660=87, srv1510458907=21, srv1925851644=44, srv1770285521=38, srv1952167075=45, srv929223649=95, srv1086612597=3, srv2100525507=55, srv812818676=92, srv1280681075=10, srv457347245=63, srv1195046023=6, srv957582320=97, srv1331477267=12, srv632138935=75, srv813786702=93, srv747221021=88, srv686463410=82, srv564094276=72, srv1612617850=29, srv703632214=84, srv673867203=81, srv707163970=85, srv1367025230=13, srv1832171415=42, srv1680382957=32, srv64340107=78, srv2010042268=50, srv515788123=67, srv1499368511=19, srv527266708=69, srv757792109=89, srv949499240=96, srv367272591=61, srv1261291499=9, srv195863619=47, srv1010433963=0, srv159391221=26, srv208631344=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:53:26,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:53:26,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:53:26,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 16:53:26,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 16:53:26,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 16:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 16:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 16:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 16:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 16:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 16:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 16:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 16:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 16:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 16:53:26,885 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:26,886 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 16:53:26,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 16:53:27,728 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1042 ms to try 5000 different iterations. Found a solution that moves 4626 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.19189652369953725. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03973506224315955, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05586780934681546, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.514, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03973506224315955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18766666666666668, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24958333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:27,742 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv195891643:81 , srv1203836290:83 , srv1680382957:83 , srv1745496204:83 , srv1773373249:83 , srv1925851644:83 , srv2075292639:83 , srv281718860:83 , srv527266708:83 , srv652864000:83 , srv673867203:83 , srv757792109:83 , srv812818676:83 , srv1127839345:84 , srv1280681075:84 , srv1331477267:84 , srv1570879323:84 , srv1591762379:84 , srv2125757423:84 , srv367272591:84 , srv694581975:84 , srv801129063:84 , srv949499240:84 , srv957582320:84 , srv1082245196:85 , srv1600256608:85 , srv1672626812:85 , srv1761579789:85 , srv20368600:85 , srv457347245:85 , srv501963783:85 , srv564094276:85 , srv632138935:85 , srv64340107:85 , srv738094660:85 , srv929223649:85 , srv992747743:85 , srv1168756977:86 , srv1459004596:86 , srv149586711:86 , srv1499368511:86 , srv1589842295:86 , srv17471883:86 , srv1812224273:86 , srv1832171415:86 , srv1858838724:86 , srv1952167075:86 , srv1957986348:86 , srv1997099605:86 , srv2100525507:86 , srv40092058:86 , srv515788123:86 , srv557002828:86 , srv667254083:86 , srv703632214:86 , srv707163970:86 , srv792074520:86 , srv992917975:86 , srv1261291499:87 , srv1390298633:87 , srv1510325172:87 , srv1551047654:87 , srv1612617850:87 , srv168414492:87 , srv1770285521:87 , srv2017970305:87 , srv280736633:87 , srv513371534:87 , srv524480463:87 , srv56019388:87 , srv619456630:87 , srv1023985039:88 , srv1195046023:88 , srv1367025230:88 , srv1627567214:88 , srv1729176716:88 , srv1798990410:88 , srv195863619:88 , srv597007943:88 , srv1086612597:89 , srv1475623699:89 , srv1510458907:89 , srv2010042268:89 , srv208631344:89 , srv476097709:89 , srv635845228:89 , srv686463410:89 , srv813786702:89 , srv825854433:89 , srv1010433963:90 , srv1426403133:90 , srv32320901:90 , srv635451662:90 , srv1285759486:91 , srv33714291:91 , srv159391221:92 , srv1609151571:92 , srv747221021:95 , srv1229468118:99 , srv71802562:443 } 2023-05-15 16:53:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:53:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2017970305=51, srv1426403133=15, srv280736633=57, srv1997099605=49, srv1127839345=4, srv168414492=33, srv1082245196=2, srv1591762379=25, srv1729176716=34, srv32320901=59, srv501963783=65, srv56019388=71, srv1957986348=46, srv1570879323=23, srv1229468118=8, srv33714291=60, srv1168756977=5, srv1672626812=31, srv1390298633=14, srv1023985039=1, srv20368600=52, srv792074520=90, srv801129063=91, srv825854433=94, srv557002828=70, srv71802562=86, srv1203836290=7, srv195891643=48, srv992917975=99, srv1609151571=28, srv281718860=58, srv1773373249=39, srv1510325172=20, srv1589842295=24, srv694581975=83, srv476097709=64, srv652864000=79, srv1285759486=11, srv1475623699=17, srv635845228=77, srv40092058=62, srv1798990410=40, srv1761579789=37, srv667254083=80, srv1858838724=43, srv2075292639=53, srv2125757423=56, srv17471883=36, srv1745496204=35, srv619456630=74, srv513371534=66, srv524480463=68, srv597007943=73, srv1812224273=41, srv1600256608=27, srv1459004596=16, srv992747743=98, srv635451662=76, srv1551047654=22, srv149586711=18, srv1627567214=30, srv738094660=87, srv1510458907=21, srv1925851644=44, srv1770285521=38, srv1952167075=45, srv929223649=95, srv1086612597=3, srv2100525507=55, srv812818676=92, srv1280681075=10, srv457347245=63, srv1195046023=6, srv957582320=97, srv1331477267=12, srv632138935=75, srv813786702=93, srv747221021=88, srv686463410=82, srv564094276=72, srv1612617850=29, srv703632214=84, srv673867203=81, srv707163970=85, srv1367025230=13, srv1832171415=42, srv1680382957=32, srv64340107=78, srv2010042268=50, srv515788123=67, srv1499368511=19, srv527266708=69, srv757792109=89, srv949499240=96, srv367272591=61, srv1261291499=9, srv195863619=47, srv1010433963=0, srv159391221=26, srv208631344=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:53:27,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:53:27,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:53:27,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:53:27,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:53:27,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:53:27,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:53:27,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 16:53:27,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 16:53:27,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 16:53:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 16:53:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 16:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 16:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 16:53:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 16:53:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 16:53:27,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 16:53:27,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03973506224315955, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05586780934681546, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03973506224315955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18766666666666668, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24958333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:27,944 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 16:53:27,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.19186413175521033, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03973506224315955, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05586780934681546, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03973506224315955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18766666666666668, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24958333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 16:53:28,268 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 449 ms to try 5000 different iterations. Found a solution that moves 2664 regions; Going from a computed imbalance of 0.19186413175521033 to a new imbalance of 0.015973485059215666. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02229449526826614); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035574489907952284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02229449526826614); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:28,278 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1229468118:74 , srv747221021:78 , srv1285759486:80 , srv1010433963:81 , srv1609151571:82 , srv686463410:82 , srv1086612597:83 , srv1168756977:83 , srv1510325172:83 , srv1510458907:83 , srv1551047654:83 , srv159391221:83 , srv1612617850:83 , srv195863619:83 , srv2017970305:83 , srv208631344:83 , srv280736633:83 , srv635451662:83 , srv635845228:83 , srv1023985039:84 , srv1195046023:84 , srv1367025230:84 , srv1390298633:84 , srv1459004596:84 , srv1761579789:84 , srv1770285521:84 , srv1798990410:84 , srv1997099605:84 , srv2010042268:84 , srv2100525507:84 , srv32320901:84 , srv524480463:84 , srv557002828:84 , srv597007943:84 , srv619456630:84 , srv703632214:84 , srv813786702:84 , srv825854433:84 , srv992917975:84 , srv1280681075:85 , srv1426403133:85 , srv1475623699:85 , srv149586711:85 , srv1627567214:85 , srv168414492:85 , srv1729176716:85 , srv1832171415:85 , srv1858838724:85 , srv33714291:85 , srv367272591:85 , srv476097709:85 , srv501963783:85 , srv513371534:85 , srv667254083:85 , srv929223649:85 , srv17471883:86 , srv457347245:86 , srv515788123:86 , srv632138935:86 , srv1127839345:87 , srv1203836290:87 , srv1261291499:87 , srv1499368511:87 , srv1672626812:87 , srv1773373249:87 , srv1812224273:87 , srv20368600:87 , srv40092058:87 , srv56019388:87 , srv673867203:87 , srv738094660:87 , srv949499240:87 , srv992747743:87 , srv1082245196:88 , srv1589842295:88 , srv1591762379:88 , srv1600256608:88 , srv1680382957:88 , srv1952167075:88 , srv564094276:88 , srv64340107:88 , srv652864000:88 , srv694581975:88 , srv707163970:88 , srv792074520:88 , srv801129063:88 , srv957582320:88 , srv1331477267:89 , srv1745496204:89 , srv1957986348:89 , srv2125757423:89 , srv281718860:89 , srv527266708:89 , srv1925851644:90 , srv812818676:90 , srv1570879323:91 , srv195891643:91 , srv2075292639:91 , srv757792109:94 , srv71802562:529 } 2023-05-15 16:53:28,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:53:28,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2017970305=51, srv1426403133=15, srv280736633=57, srv1997099605=49, srv1127839345=4, srv168414492=33, srv1082245196=2, srv1591762379=25, srv1729176716=34, srv32320901=59, srv501963783=65, srv56019388=71, srv1957986348=46, srv1570879323=23, srv1229468118=8, srv33714291=60, srv1168756977=5, srv1672626812=31, srv1390298633=14, srv1023985039=1, srv20368600=52, srv792074520=90, srv801129063=91, srv825854433=94, srv557002828=70, srv71802562=86, srv1203836290=7, srv195891643=48, srv992917975=99, srv1609151571=28, srv281718860=58, srv1773373249=39, srv1510325172=20, srv1589842295=24, srv694581975=83, srv476097709=64, srv652864000=79, srv1285759486=11, srv1475623699=17, srv635845228=77, srv40092058=62, srv1798990410=40, srv1761579789=37, srv667254083=80, srv1858838724=43, srv2075292639=53, srv2125757423=56, srv17471883=36, srv1745496204=35, srv619456630=74, srv513371534=66, srv524480463=68, srv597007943=73, srv1812224273=41, srv1600256608=27, srv1459004596=16, srv992747743=98, srv635451662=76, srv1551047654=22, srv149586711=18, srv1627567214=30, srv738094660=87, srv1510458907=21, srv1925851644=44, srv1770285521=38, srv1952167075=45, srv929223649=95, srv1086612597=3, srv2100525507=55, srv812818676=92, srv1280681075=10, srv457347245=63, srv1195046023=6, srv957582320=97, srv1331477267=12, srv632138935=75, srv813786702=93, srv747221021=88, srv686463410=82, srv564094276=72, srv1612617850=29, srv703632214=84, srv673867203=81, srv707163970=85, srv1367025230=13, srv1832171415=42, srv1680382957=32, srv64340107=78, srv2010042268=50, srv515788123=67, srv1499368511=19, srv527266708=69, srv757792109=89, srv949499240=96, srv367272591=61, srv1261291499=9, srv195863619=47, srv1010433963=0, srv159391221=26, srv208631344=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 16:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 16:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 16:53:28,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 16:53:28,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 16:53:28,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 16:53:28,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 16:53:28,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 16:53:28,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 16:53:28,424 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02229449526826614); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035574489907952284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02229449526826614); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:28,424 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 16:53:28,424 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015954831332521572, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02229449526826614); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035574489907952284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02229449526826614); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 16:53:28,726 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 406 ms to try 5000 different iterations. Found a solution that moves 996 regions; Going from a computed imbalance of 0.015954831332521572 to a new imbalance of 6.259265948144426E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006765079824034915); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0069307946660236135); MoveCostFunction : (multiplier=7.0, imbalance=0.11066666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006765079824034915); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0028333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:28,728 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv757792109:80 , srv1957986348:81 , srv1426403133:82 , srv1609151571:82 , srv707163970:82 , srv1570879323:83 , srv159391221:83 , srv1952167075:83 , srv2075292639:83 , srv33714291:83 , srv1082245196:84 , srv1261291499:84 , srv1475623699:84 , srv1499368511:84 , srv1589842295:84 , srv1627567214:84 , srv1729176716:84 , srv1812224273:84 , srv2010042268:84 , srv2125757423:84 , srv32320901:84 , srv40092058:84 , srv476097709:84 , srv56019388:84 , srv564094276:84 , srv64340107:84 , srv792074520:84 , srv812818676:84 , srv825854433:84 , srv1023985039:85 , srv1086612597:85 , srv1195046023:85 , srv1229468118:85 , srv1331477267:85 , srv1367025230:85 , srv1510458907:85 , srv1600256608:85 , srv17471883:85 , srv1925851644:85 , srv208631344:85 , srv281718860:85 , srv513371534:85 , srv515788123:85 , srv635451662:85 , srv635845228:85 , srv738094660:85 , srv747221021:85 , srv813786702:85 , srv957582320:85 , srv1010433963:86 , srv1285759486:86 , srv1390298633:86 , srv149586711:86 , srv1591762379:86 , srv1672626812:86 , srv1680382957:86 , srv168414492:86 , srv1745496204:86 , srv1770285521:86 , srv1798990410:86 , srv1832171415:86 , srv1858838724:86 , srv195891643:86 , srv20368600:86 , srv2100525507:86 , srv524480463:86 , srv527266708:86 , srv597007943:86 , srv619456630:86 , srv694581975:86 , srv801129063:86 , srv992747743:86 , srv1127839345:87 , srv1510325172:87 , srv1551047654:87 , srv1612617850:87 , srv1773373249:87 , srv195863619:87 , srv1997099605:87 , srv2017970305:87 , srv280736633:87 , srv457347245:87 , srv501963783:87 , srv557002828:87 , srv632138935:87 , srv652864000:87 , srv667254083:87 , srv673867203:87 , srv686463410:87 , srv703632214:87 , srv949499240:87 , srv1168756977:88 , srv1203836290:88 , srv1280681075:88 , srv1459004596:88 , srv1761579789:88 , srv929223649:88 , srv992917975:88 , srv367272591:89 , srv71802562:546 } 2023-05-15 16:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2017970305=51, srv1426403133=15, srv280736633=57, srv1997099605=49, srv1127839345=4, srv168414492=33, srv1082245196=2, srv1591762379=25, srv1729176716=34, srv32320901=59, srv501963783=65, srv56019388=71, srv1957986348=46, srv1570879323=23, srv1229468118=8, srv33714291=60, srv1168756977=5, srv1672626812=31, srv1390298633=14, srv1023985039=1, srv20368600=52, srv792074520=90, srv801129063=91, srv825854433=94, srv557002828=70, srv71802562=86, srv1203836290=7, srv195891643=48, srv992917975=99, srv1609151571=28, srv281718860=58, srv1773373249=39, srv1510325172=20, srv1589842295=24, srv694581975=83, srv476097709=64, srv652864000=79, srv1285759486=11, srv1475623699=17, srv635845228=77, srv40092058=62, srv1798990410=40, srv1761579789=37, srv667254083=80, srv1858838724=43, srv2075292639=53, srv2125757423=56, srv17471883=36, srv1745496204=35, srv619456630=74, srv513371534=66, srv524480463=68, srv597007943=73, srv1812224273=41, srv1600256608=27, srv1459004596=16, srv992747743=98, srv635451662=76, srv1551047654=22, srv149586711=18, srv1627567214=30, srv738094660=87, srv1510458907=21, srv1925851644=44, srv1770285521=38, srv1952167075=45, srv929223649=95, srv1086612597=3, srv2100525507=55, srv812818676=92, srv1280681075=10, srv457347245=63, srv1195046023=6, srv957582320=97, srv1331477267=12, srv632138935=75, srv813786702=93, srv747221021=88, srv686463410=82, srv564094276=72, srv1612617850=29, srv703632214=84, srv673867203=81, srv707163970=85, srv1367025230=13, srv1832171415=42, srv1680382957=32, srv64340107=78, srv2010042268=50, srv515788123=67, srv1499368511=19, srv527266708=69, srv757792109=89, srv949499240=96, srv367272591=61, srv1261291499=9, srv195863619=47, srv1010433963=0, srv159391221=26, srv208631344=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 16:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 16:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 16:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 16:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 16:53:28,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 16:53:28,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 16:53:28,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 16:53:28,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 16:53:28,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 16:53:28,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 16:53:28,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 16:53:28,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 16:53:28,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 16:53:28,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 16:53:28,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 16:53:28,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 16:53:28,936 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006765079824034915); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0069307946660236135); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006765079824034915); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0028333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:28,937 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 16:53:28,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.189524537531366E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006765079824034915); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0069307946660236135); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006765079824034915); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0028333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 16:53:29,257 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 449 ms to try 5000 different iterations. Found a solution that moves 230 regions; Going from a computed imbalance of 6.189524537531366E-4 to a new imbalance of 1.1733461329801921E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.025555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:29,258 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1261291499:83 , srv159391221:83 , srv33714291:83 , srv1127839345:84 , srv1203836290:84 , srv1229468118:84 , srv1459004596:84 , srv1475623699:84 , srv1551047654:84 , srv1570879323:84 , srv1589842295:84 , srv1591762379:84 , srv1600256608:84 , srv1672626812:84 , srv168414492:84 , srv1798990410:84 , srv1925851644:84 , srv195863619:84 , srv195891643:84 , srv2010042268:84 , srv20368600:84 , srv32320901:84 , srv367272591:84 , srv527266708:84 , srv597007943:84 , srv619456630:84 , srv632138935:84 , srv635451662:84 , srv635845228:84 , srv652864000:84 , srv667254083:84 , srv686463410:84 , srv694581975:84 , srv747221021:84 , srv792074520:84 , srv801129063:84 , srv813786702:84 , srv929223649:84 , srv949499240:84 , srv992747743:84 , srv992917975:84 , srv1010433963:85 , srv1023985039:85 , srv1082245196:85 , srv1086612597:85 , srv1168756977:85 , srv1195046023:85 , srv1280681075:85 , srv1285759486:85 , srv1331477267:85 , srv1367025230:85 , srv1390298633:85 , srv1426403133:85 , srv149586711:85 , srv1499368511:85 , srv1510325172:85 , srv1510458907:85 , srv1612617850:85 , srv1627567214:85 , srv1680382957:85 , srv1729176716:85 , srv1745496204:85 , srv17471883:85 , srv1761579789:85 , srv1770285521:85 , srv1773373249:85 , srv1812224273:85 , srv1832171415:85 , srv1858838724:85 , srv1952167075:85 , srv1997099605:85 , srv2017970305:85 , srv2075292639:85 , srv208631344:85 , srv2125757423:85 , srv280736633:85 , srv281718860:85 , srv40092058:85 , srv457347245:85 , srv476097709:85 , srv501963783:85 , srv513371534:85 , srv515788123:85 , srv524480463:85 , srv557002828:85 , srv56019388:85 , srv564094276:85 , srv64340107:85 , srv673867203:85 , srv703632214:85 , srv738094660:85 , srv757792109:85 , srv812818676:85 , srv825854433:85 , srv957582320:85 , srv1957986348:86 , srv2100525507:86 , srv707163970:86 , srv1609151571:87 , srv71802562:624 } 2023-05-15 16:53:29,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:53:29,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2017970305=51, srv1426403133=15, srv280736633=57, srv1997099605=49, srv1127839345=4, srv168414492=33, srv1082245196=2, srv1591762379=25, srv1729176716=34, srv32320901=59, srv501963783=65, srv56019388=71, srv1957986348=46, srv1570879323=23, srv1229468118=8, srv33714291=60, srv1168756977=5, srv1672626812=31, srv1390298633=14, srv1023985039=1, srv20368600=52, srv792074520=90, srv801129063=91, srv825854433=94, srv557002828=70, srv71802562=86, srv1203836290=7, srv195891643=48, srv992917975=99, srv1609151571=28, srv281718860=58, srv1773373249=39, srv1510325172=20, srv1589842295=24, srv694581975=83, srv476097709=64, srv652864000=79, srv1285759486=11, srv1475623699=17, srv635845228=77, srv40092058=62, srv1798990410=40, srv1761579789=37, srv667254083=80, srv1858838724=43, srv2075292639=53, srv2125757423=56, srv17471883=36, srv1745496204=35, srv619456630=74, srv513371534=66, srv524480463=68, srv597007943=73, srv1812224273=41, srv1600256608=27, srv1459004596=16, srv992747743=98, srv635451662=76, srv1551047654=22, srv149586711=18, srv1627567214=30, srv738094660=87, srv1510458907=21, srv1925851644=44, srv1770285521=38, srv1952167075=45, srv929223649=95, srv1086612597=3, srv2100525507=55, srv812818676=92, srv1280681075=10, srv457347245=63, srv1195046023=6, srv957582320=97, srv1331477267=12, srv632138935=75, srv813786702=93, srv747221021=88, srv686463410=82, srv564094276=72, srv1612617850=29, srv703632214=84, srv673867203=81, srv707163970=85, srv1367025230=13, srv1832171415=42, srv1680382957=32, srv64340107=78, srv2010042268=50, srv515788123=67, srv1499368511=19, srv527266708=69, srv757792109=89, srv949499240=96, srv367272591=61, srv1261291499=9, srv195863619=47, srv1010433963=0, srv159391221=26, srv208631344=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:53:29,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:53:29,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:53:29,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:53:29,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:53:29,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 16:53:29,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 16:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 16:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 16:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 16:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 16:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 16:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 16:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 16:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 16:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 16:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 16:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 16:53:29,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 16:53:29,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 16:53:29,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 16:53:29,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 16:53:29,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 16:53:29,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 16:53:29,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 16:53:29,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 16:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 16:53:29,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 16:53:29,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 16:53:29,442 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0122966908014432E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:29,442 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1261291499:83 , srv159391221:83 , srv33714291:83 , srv1127839345:84 , srv1203836290:84 , srv1229468118:84 , srv1459004596:84 , srv1475623699:84 , srv1551047654:84 , srv1570879323:84 , srv1589842295:84 , srv1591762379:84 , srv1600256608:84 , srv1672626812:84 , srv168414492:84 , srv1798990410:84 , srv1925851644:84 , srv195863619:84 , srv195891643:84 , srv2010042268:84 , srv20368600:84 , srv32320901:84 , srv367272591:84 , srv527266708:84 , srv597007943:84 , srv619456630:84 , srv632138935:84 , srv635451662:84 , srv635845228:84 , srv652864000:84 , srv667254083:84 , srv686463410:84 , srv694581975:84 , srv747221021:84 , srv792074520:84 , srv801129063:84 , srv813786702:84 , srv929223649:84 , srv949499240:84 , srv992747743:84 , srv992917975:84 , srv1010433963:85 , srv1023985039:85 , srv1082245196:85 , srv1086612597:85 , srv1168756977:85 , srv1195046023:85 , srv1280681075:85 , srv1285759486:85 , srv1331477267:85 , srv1367025230:85 , srv1390298633:85 , srv1426403133:85 , srv149586711:85 , srv1499368511:85 , srv1510325172:85 , srv1510458907:85 , srv1612617850:85 , srv1627567214:85 , srv1680382957:85 , srv1729176716:85 , srv1745496204:85 , srv17471883:85 , srv1761579789:85 , srv1770285521:85 , srv1773373249:85 , srv1812224273:85 , srv1832171415:85 , srv1858838724:85 , srv1952167075:85 , srv1997099605:85 , srv2017970305:85 , srv2075292639:85 , srv208631344:85 , srv2125757423:85 , srv280736633:85 , srv281718860:85 , srv40092058:85 , srv457347245:85 , srv476097709:85 , srv501963783:85 , srv513371534:85 , srv515788123:85 , srv524480463:85 , srv557002828:85 , srv56019388:85 , srv564094276:85 , srv64340107:85 , srv673867203:85 , srv703632214:85 , srv738094660:85 , srv757792109:85 , srv812818676:85 , srv825854433:85 , srv957582320:85 , srv1957986348:86 , srv2100525507:86 , srv707163970:86 , srv1609151571:87 , srv71802562:624 } 2023-05-15 16:53:29,481 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=274 (was 274), ProcessCount=173 (was 173), AvailableMemoryMB=13817 (was 14666) 2023-05-15 16:53:29,503 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=274, ProcessCount=173, AvailableMemoryMB=13812 2023-05-15 16:53:29,504 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 16:53:29,504 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-15 16:53:29,521 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010253524:84 , srv1131879264:84 , srv119176905:84 , srv1210137747:84 , srv1250952267:84 , srv1326498178:84 , srv1339390569:84 , srv1361687550:84 , srv1506854608:84 , srv1623897059:84 , srv1802985546:84 , srv1993629483:84 , srv2028573326:84 , srv2047416151:84 , srv2115827332:84 , srv2129644307:84 , srv23884745:84 , srv304538798:84 , srv329427663:84 , srv345401243:84 , srv395928921:84 , srv518351765:84 , srv538290548:84 , srv538445874:84 , srv670133858:84 , srv736215820:84 , srv762763681:84 , srv904114528:84 , srv970406141:84 , srv986989180:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-15 16:53:29,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:53:29,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv119176905=2, srv762763681=26, srv1010253524=0, srv2047416151=13, srv345401243=19, srv329427663=18, srv986989180=29, srv1250952267=4, srv2115827332=14, srv538445874=23, srv395928921=20, srv670133858=24, srv736215820=25, srv1339390569=6, srv1506854608=8, srv1623897059=9, srv1131879264=1, srv1210137747=3, srv1993629483=11, srv518351765=21, srv1802985546=10, srv1326498178=5, srv304538798=17, srv2129644307=15, srv23884745=16, srv904114528=27, srv970406141=28, srv1361687550=7, srv2028573326=12, srv538290548=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-15 16:53:29,564 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1248699822481794, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:53:29,565 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912586504500963, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1248699822481794, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-15 16:55:29,543 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6808127 different iterations. Found a solution that moves 1842 regions; Going from a computed imbalance of 0.9912586504500963 to a new imbalance of 4.2993198912066E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:55:29,549 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1010253524:90 , srv1131879264:90 , srv119176905:90 , srv1210137747:90 , srv1250952267:90 , srv1326498178:90 , srv1339390569:90 , srv1361687550:90 , srv1506854608:90 , srv1623897059:90 , srv1802985546:90 , srv1993629483:90 , srv2028573326:90 , srv2047416151:90 , srv2115827332:90 , srv2129644307:90 , srv23884745:90 , srv304538798:90 , srv329427663:90 , srv345401243:90 , srv395928921:90 , srv518351765:90 , srv538290548:90 , srv538445874:90 , srv670133858:90 , srv736215820:90 , srv762763681:90 , srv904114528:90 , srv970406141:90 , srv986989180:90 } 2023-05-15 16:55:29,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 16:55:29,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv119176905=2, srv762763681=26, srv1010253524=0, srv2047416151=13, srv345401243=19, srv329427663=18, srv986989180=29, srv1250952267=4, srv2115827332=14, srv538445874=23, srv395928921=20, srv670133858=24, srv736215820=25, srv1339390569=6, srv1506854608=8, srv1623897059=9, srv1131879264=1, srv1210137747=3, srv1993629483=11, srv518351765=21, srv1802985546=10, srv1326498178=5, srv304538798=17, srv2129644307=15, srv23884745=16, srv904114528=27, srv970406141=28, srv1361687550=7, srv2028573326=12, srv538290548=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 16:55:29,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 16:55:29,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-15 16:55:29,567 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 16:55:29,567 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1010253524:90 , srv1131879264:90 , srv119176905:90 , srv1210137747:90 , srv1250952267:90 , srv1326498178:90 , srv1339390569:90 , srv1361687550:90 , srv1506854608:90 , srv1623897059:90 , srv1802985546:90 , srv1993629483:90 , srv2028573326:90 , srv2047416151:90 , srv2115827332:90 , srv2129644307:90 , srv23884745:90 , srv304538798:90 , srv329427663:90 , srv345401243:90 , srv395928921:90 , srv518351765:90 , srv538290548:90 , srv538445874:90 , srv670133858:90 , srv736215820:90 , srv762763681:90 , srv904114528:90 , srv970406141:90 , srv986989180:90 } 2023-05-15 16:55:29,588 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=169 (was 274), ProcessCount=169 (was 173), AvailableMemoryMB=13970 (was 13812) - AvailableMemoryMB LEAK? -