2023-10-24 17:53:26,884 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-24 17:53:27,180 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-24 17:53:27,282 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-24 17:53:27,341 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-24 17:53:27,348 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-24 17:53:27,361 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=169, AvailableMemoryMB=8473 2023-10-24 17:53:27,363 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-24 17:53:27,364 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-24 17:53:27,504 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1050833484:84 , srv1072596864:84 , srv1107502226:84 , srv1138817541:84 , srv1148932263:84 , srv1159003577:84 , srv1168949028:84 , srv1170401864:84 , srv1202114582:84 , srv121903446:84 , srv1241377500:84 , srv1253224911:84 , srv127069451:84 , srv1329931440:84 , srv1353268395:84 , srv139008636:84 , srv1393430968:84 , srv1398103414:84 , srv1408434441:84 , srv1437113798:84 , srv1440515616:84 , srv1441397235:84 , srv1441741628:84 , srv1444440144:84 , srv1464628303:84 , srv1467690478:84 , srv1478176340:84 , srv1491380947:84 , srv1502574985:84 , srv153444522:84 , srv1583020570:84 , srv1609684384:84 , srv1612800911:84 , srv1639568859:84 , srv1644730159:84 , srv1671926698:84 , srv1678091262:84 , srv168032846:84 , srv168993262:84 , srv1690586985:84 , srv1694167812:84 , srv1728053811:84 , srv1765844386:84 , srv1768562234:84 , srv1788135002:84 , srv1793231635:84 , srv1795704105:84 , srv1814175923:84 , srv1822836179:84 , srv1849693652:84 , srv1867351047:84 , srv1894120202:84 , srv1911623048:84 , srv1927522878:84 , srv198380182:84 , srv1992104131:84 , srv1995103527:84 , srv2003549916:84 , srv2008696831:84 , srv2041799225:84 , srv2043549199:84 , srv207540720:84 , srv2129463572:84 , srv255730540:84 , srv262237444:84 , srv266705009:84 , srv294292153:84 , srv337231076:84 , srv342491125:84 , srv34429903:84 , srv351156577:84 , srv369159706:84 , srv402315486:84 , srv408012648:84 , srv417677520:84 , srv418466136:84 , srv42524708:84 , srv428614230:84 , srv450574713:84 , srv477066171:84 , srv499442939:84 , srv499812283:84 , srv50101477:84 , srv545851645:84 , srv587654233:84 , srv590627876:84 , srv678397925:84 , srv714841081:84 , srv740983790:84 , srv750606200:84 , srv754020213:84 , srv784528201:84 , srv798579629:84 , srv862630396:84 , srv899427774:84 , srv922220509:84 , srv976750421:84 , srv980750520:84 , srv994644735:84 , srv2047121235:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-24 17:53:27,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:53:27,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464628303=24, srv1911623048=52, srv34429903=70, srv899427774=95, srv1241377500=10, srv1788135002=44, srv1444440144=23, srv1491380947=27, srv1329931440=13, srv168993262=38, srv1822836179=48, srv42524708=77, srv50101477=83, srv750606200=90, srv1467690478=25, srv784528201=92, srv740983790=89, srv262237444=65, srv1148932263=4, srv1644730159=34, srv255730540=64, srv450574713=79, srv2041799225=59, srv1765844386=42, srv1478176340=26, srv1138817541=3, srv342491125=69, srv418466136=76, srv1814175923=47, srv1202114582=8, srv994644735=99, srv1671926698=35, srv545851645=84, srv2129463572=63, srv1678091262=36, srv168032846=37, srv976750421=97, srv1441741628=22, srv1849693652=49, srv1253224911=11, srv1168949028=6, srv1437113798=19, srv754020213=91, srv207540720=62, srv798579629=93, srv1440515616=20, srv862630396=94, srv1639568859=33, srv1728053811=41, srv294292153=67, srv1609684384=31, srv2003549916=57, srv1441397235=21, srv1995103527=56, srv2043549199=60, srv127069451=12, srv1502574985=28, srv1894120202=51, srv1690586985=39, srv1170401864=7, srv266705009=66, srv417677520=75, srv1795704105=46, srv1793231635=45, srv1612800911=32, srv922220509=96, srv2047121235=61, srv1159003577=5, srv402315486=73, srv678397925=87, srv139008636=15, srv408012648=74, srv714841081=88, srv1393430968=16, srv351156577=71, srv590627876=86, srv1867351047=50, srv1050833484=0, srv1583020570=30, srv1927522878=53, srv1992104131=55, srv1408434441=18, srv198380182=54, srv337231076=68, srv428614230=78, srv499812283=82, srv499442939=81, srv1107502226=2, srv1072596864=1, srv369159706=72, srv1768562234=43, srv1353268395=14, srv2008696831=58, srv1694167812=40, srv477066171=80, srv587654233=85, srv121903446=9, srv1398103414=17, srv153444522=29, srv980750520=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:53:27,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:53:27,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:53:27,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:53:27,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 17:53:27,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 17:53:27,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 17:53:27,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 17:53:27,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 17:53:27,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 17:53:27,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 17:53:27,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 17:53:27,901 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:27,901 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 17:53:27,901 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 17:53:28,305 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 668 ms to try 5000 different iterations. Found a solution that moves 4680 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18314711832877278. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04120851748069478, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06049181150584978, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.52, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04120851748069478, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1785, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:28,316 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv127069451:81 , srv1995103527:81 , srv255730540:81 , srv337231076:81 , srv980750520:81 , srv1170401864:82 , srv1408434441:82 , srv1671926698:82 , srv750606200:82 , srv1444440144:83 , srv1609684384:83 , srv168993262:83 , srv1694167812:83 , srv1768562234:83 , srv1814175923:83 , srv2043549199:83 , srv428614230:83 , srv784528201:83 , srv862630396:83 , srv899427774:83 , srv1202114582:84 , srv1253224911:84 , srv1644730159:84 , srv1788135002:84 , srv1822836179:84 , srv1107502226:85 , srv1159003577:85 , srv121903446:85 , srv1393430968:85 , srv1491380947:85 , srv1678091262:85 , srv1690586985:85 , srv1765844386:85 , srv198380182:85 , srv2129463572:85 , srv266705009:85 , srv351156577:85 , srv369159706:85 , srv402315486:85 , srv408012648:85 , srv417677520:85 , srv418466136:85 , srv1241377500:86 , srv1478176340:86 , srv153444522:86 , srv1728053811:86 , srv1894120202:86 , srv2008696831:86 , srv207540720:86 , srv42524708:86 , srv477066171:86 , srv499442939:86 , srv678397925:86 , srv714841081:86 , srv798579629:86 , srv1072596864:87 , srv1138817541:87 , srv1148932263:87 , srv139008636:87 , srv1441741628:87 , srv1639568859:87 , srv1927522878:87 , srv262237444:87 , srv450574713:87 , srv590627876:87 , srv994644735:87 , srv1398103414:88 , srv1437113798:88 , srv1612800911:88 , srv1793231635:88 , srv1795704105:88 , srv1992104131:88 , srv2003549916:88 , srv2041799225:88 , srv342491125:88 , srv34429903:88 , srv587654233:88 , srv740983790:88 , srv1440515616:89 , srv1583020570:89 , srv1849693652:89 , srv1911623048:89 , srv50101477:89 , srv1050833484:90 , srv1353268395:90 , srv1464628303:90 , srv1502574985:90 , srv1867351047:90 , srv499812283:90 , srv545851645:90 , srv922220509:90 , srv1168949028:91 , srv1329931440:91 , srv1441397235:91 , srv168032846:91 , srv1467690478:92 , srv294292153:92 , srv976750421:94 , srv754020213:95 , srv2047121235:456 } 2023-10-24 17:53:28,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:53:28,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464628303=24, srv1911623048=52, srv34429903=70, srv899427774=95, srv1241377500=10, srv1788135002=44, srv1444440144=23, srv1491380947=27, srv1329931440=13, srv168993262=38, srv1822836179=48, srv42524708=77, srv50101477=83, srv750606200=90, srv1467690478=25, srv784528201=92, srv740983790=89, srv262237444=65, srv1148932263=4, srv1644730159=34, srv255730540=64, srv450574713=79, srv2041799225=59, srv1765844386=42, srv1478176340=26, srv1138817541=3, srv342491125=69, srv418466136=76, srv1814175923=47, srv1202114582=8, srv994644735=99, srv1671926698=35, srv545851645=84, srv2129463572=63, srv1678091262=36, srv168032846=37, srv976750421=97, srv1441741628=22, srv1849693652=49, srv1253224911=11, srv1168949028=6, srv1437113798=19, srv754020213=91, srv207540720=62, srv798579629=93, srv1440515616=20, srv862630396=94, srv1639568859=33, srv1728053811=41, srv294292153=67, srv1609684384=31, srv2003549916=57, srv1441397235=21, srv1995103527=56, srv2043549199=60, srv127069451=12, srv1502574985=28, srv1894120202=51, srv1690586985=39, srv1170401864=7, srv266705009=66, srv417677520=75, srv1795704105=46, srv1793231635=45, srv1612800911=32, srv922220509=96, srv2047121235=61, srv1159003577=5, srv402315486=73, srv678397925=87, srv139008636=15, srv408012648=74, srv714841081=88, srv1393430968=16, srv351156577=71, srv590627876=86, srv1867351047=50, srv1050833484=0, srv1583020570=30, srv1927522878=53, srv1992104131=55, srv1408434441=18, srv198380182=54, srv337231076=68, srv428614230=78, srv499812283=82, srv499442939=81, srv1107502226=2, srv1072596864=1, srv369159706=72, srv1768562234=43, srv1353268395=14, srv2008696831=58, srv1694167812=40, srv477066171=80, srv587654233=85, srv121903446=9, srv1398103414=17, srv153444522=29, srv980750520=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:53:28,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:53:28,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:53:28,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:53:28,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:53:28,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:53:28,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:53:28,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:53:28,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:53:28,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:53:28,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:53:28,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:53:28,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:53:28,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:53:28,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:53:28,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 17:53:28,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 17:53:28,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 17:53:28,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 17:53:28,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 17:53:28,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 17:53:28,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 17:53:28,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 17:53:28,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 17:53:28,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 17:53:28,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 17:53:28,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 17:53:28,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 17:53:28,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 17:53:28,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04120851748069478, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06049181150584978, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04120851748069478, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1785, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:28,529 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 17:53:28,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18311434826836426, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04120851748069478, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06049181150584978, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04120851748069478, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1785, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 17:53:28,832 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 2661 regions; Going from a computed imbalance of 0.18311434826836426 to a new imbalance of 0.016230884448346077. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020714870831128007); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03111832528190051, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2956666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020714870831128007); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:28,843 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv754020213:78 , srv1329931440:79 , srv1502574985:80 , srv294292153:80 , srv976750421:80 , srv1353268395:81 , srv168032846:81 , srv499812283:81 , srv1050833484:82 , srv1441397235:82 , srv1464628303:82 , srv1467690478:82 , srv34429903:82 , srv50101477:82 , srv922220509:82 , srv139008636:83 , srv1398103414:83 , srv1440515616:83 , srv1583020570:83 , srv1849693652:83 , srv2003549916:83 , srv342491125:83 , srv42524708:83 , srv545851645:83 , srv740983790:83 , srv1072596864:84 , srv1138817541:84 , srv1168949028:84 , srv1795704105:84 , srv1867351047:84 , srv1911623048:84 , srv1927522878:84 , srv450574713:84 , srv587654233:84 , srv590627876:84 , srv994644735:84 , srv1241377500:85 , srv1393430968:85 , srv153444522:85 , srv1612800911:85 , srv1728053811:85 , srv1765844386:85 , srv1793231635:85 , srv1992104131:85 , srv2008696831:85 , srv2041799225:85 , srv262237444:85 , srv499442939:85 , srv714841081:85 , srv1107502226:86 , srv121903446:86 , srv1437113798:86 , srv1441741628:86 , srv1478176340:86 , srv1639568859:86 , srv1644730159:86 , srv1678091262:86 , srv1894120202:86 , srv266705009:86 , srv477066171:86 , srv1148932263:87 , srv1159003577:87 , srv1202114582:87 , srv207540720:87 , srv351156577:87 , srv369159706:87 , srv402315486:87 , srv408012648:87 , srv1408434441:88 , srv1690586985:88 , srv198380182:88 , srv2129463572:88 , srv417677520:88 , srv1170401864:89 , srv1253224911:89 , srv1444440144:89 , srv1491380947:89 , srv1768562234:89 , srv1822836179:89 , srv418466136:89 , srv750606200:89 , srv784528201:89 , srv127069451:90 , srv1609684384:90 , srv1694167812:90 , srv1788135002:90 , srv2043549199:90 , srv678397925:90 , srv862630396:90 , srv899427774:90 , srv1671926698:91 , srv168993262:91 , srv1814175923:91 , srv1995103527:91 , srv255730540:91 , srv337231076:91 , srv980750520:91 , srv428614230:92 , srv798579629:93 , srv2047121235:502 } 2023-10-24 17:53:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:53:28,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464628303=24, srv1911623048=52, srv34429903=70, srv899427774=95, srv1241377500=10, srv1788135002=44, srv1444440144=23, srv1491380947=27, srv1329931440=13, srv168993262=38, srv1822836179=48, srv42524708=77, srv50101477=83, srv750606200=90, srv1467690478=25, srv784528201=92, srv740983790=89, srv262237444=65, srv1148932263=4, srv1644730159=34, srv255730540=64, srv450574713=79, srv2041799225=59, srv1765844386=42, srv1478176340=26, srv1138817541=3, srv342491125=69, srv418466136=76, srv1814175923=47, srv1202114582=8, srv994644735=99, srv1671926698=35, srv545851645=84, srv2129463572=63, srv1678091262=36, srv168032846=37, srv976750421=97, srv1441741628=22, srv1849693652=49, srv1253224911=11, srv1168949028=6, srv1437113798=19, srv754020213=91, srv207540720=62, srv798579629=93, srv1440515616=20, srv862630396=94, srv1639568859=33, srv1728053811=41, srv294292153=67, srv1609684384=31, srv2003549916=57, srv1441397235=21, srv1995103527=56, srv2043549199=60, srv127069451=12, srv1502574985=28, srv1894120202=51, srv1690586985=39, srv1170401864=7, srv266705009=66, srv417677520=75, srv1795704105=46, srv1793231635=45, srv1612800911=32, srv922220509=96, srv2047121235=61, srv1159003577=5, srv402315486=73, srv678397925=87, srv139008636=15, srv408012648=74, srv714841081=88, srv1393430968=16, srv351156577=71, srv590627876=86, srv1867351047=50, srv1050833484=0, srv1583020570=30, srv1927522878=53, srv1992104131=55, srv1408434441=18, srv198380182=54, srv337231076=68, srv428614230=78, srv499812283=82, srv499442939=81, srv1107502226=2, srv1072596864=1, srv369159706=72, srv1768562234=43, srv1353268395=14, srv2008696831=58, srv1694167812=40, srv477066171=80, srv587654233=85, srv121903446=9, srv1398103414=17, srv153444522=29, srv980750520=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:53:28,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:53:28,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:53:28,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:53:28,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:53:28,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:53:28,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:53:28,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 17:53:28,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 17:53:28,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 17:53:28,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 17:53:28,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 17:53:28,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 17:53:28,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 17:53:28,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 17:53:28,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 17:53:29,078 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020714870831128007); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03111832528190051, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020714870831128007); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:29,078 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 17:53:29,078 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016212251728100962, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020714870831128007); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03111832528190051, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020714870831128007); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 17:53:29,346 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations. Found a solution that moves 1013 regions; Going from a computed imbalance of 0.016212251728100962 to a new imbalance of 7.029586001291685E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004424760000269228); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004854793875029461); MoveCostFunction : (multiplier=7.0, imbalance=0.11255555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004424760000269228); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:29,348 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv798579629:80 , srv678397925:82 , srv1168949028:83 , srv1467690478:83 , srv428614230:83 , srv1253224911:84 , srv1437113798:84 , srv1491380947:84 , srv1609684384:84 , srv1671926698:84 , srv168993262:84 , srv1690586985:84 , srv1788135002:84 , srv1814175923:84 , srv1867351047:84 , srv1911623048:84 , srv207540720:84 , srv417677520:84 , srv418466136:84 , srv754020213:84 , srv976750421:84 , srv1050833484:85 , srv1148932263:85 , srv1159003577:85 , srv1440515616:85 , srv1441397235:85 , srv1441741628:85 , srv1444440144:85 , srv1464628303:85 , srv1612800911:85 , srv1639568859:85 , srv1694167812:85 , srv1768562234:85 , srv1793231635:85 , srv1795704105:85 , srv1822836179:85 , srv1849693652:85 , srv1894120202:85 , srv198380182:85 , srv1992104131:85 , srv1995103527:85 , srv2041799225:85 , srv2043549199:85 , srv2129463572:85 , srv255730540:85 , srv351156577:85 , srv402315486:85 , srv545851645:85 , srv862630396:85 , srv899427774:85 , srv1072596864:86 , srv1107502226:86 , srv1170401864:86 , srv1202114582:86 , srv121903446:86 , srv1241377500:86 , srv127069451:86 , srv1353268395:86 , srv1478176340:86 , srv1583020570:86 , srv1927522878:86 , srv2003549916:86 , srv2008696831:86 , srv262237444:86 , srv266705009:86 , srv337231076:86 , srv342491125:86 , srv369159706:86 , srv408012648:86 , srv450574713:86 , srv477066171:86 , srv499442939:86 , srv50101477:86 , srv587654233:86 , srv740983790:86 , srv784528201:86 , srv922220509:86 , srv980750520:86 , srv994644735:86 , srv1138817541:87 , srv1393430968:87 , srv1398103414:87 , srv1408434441:87 , srv153444522:87 , srv1678091262:87 , srv168032846:87 , srv1728053811:87 , srv294292153:87 , srv34429903:87 , srv499812283:87 , srv590627876:87 , srv714841081:87 , srv750606200:87 , srv139008636:88 , srv1502574985:88 , srv1644730159:88 , srv1765844386:88 , srv42524708:88 , srv1329931440:89 , srv2047121235:539 } 2023-10-24 17:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464628303=24, srv1911623048=52, srv34429903=70, srv899427774=95, srv1241377500=10, srv1788135002=44, srv1444440144=23, srv1491380947=27, srv1329931440=13, srv168993262=38, srv1822836179=48, srv42524708=77, srv50101477=83, srv750606200=90, srv1467690478=25, srv784528201=92, srv740983790=89, srv262237444=65, srv1148932263=4, srv1644730159=34, srv255730540=64, srv450574713=79, srv2041799225=59, srv1765844386=42, srv1478176340=26, srv1138817541=3, srv342491125=69, srv418466136=76, srv1814175923=47, srv1202114582=8, srv994644735=99, srv1671926698=35, srv545851645=84, srv2129463572=63, srv1678091262=36, srv168032846=37, srv976750421=97, srv1441741628=22, srv1849693652=49, srv1253224911=11, srv1168949028=6, srv1437113798=19, srv754020213=91, srv207540720=62, srv798579629=93, srv1440515616=20, srv862630396=94, srv1639568859=33, srv1728053811=41, srv294292153=67, srv1609684384=31, srv2003549916=57, srv1441397235=21, srv1995103527=56, srv2043549199=60, srv127069451=12, srv1502574985=28, srv1894120202=51, srv1690586985=39, srv1170401864=7, srv266705009=66, srv417677520=75, srv1795704105=46, srv1793231635=45, srv1612800911=32, srv922220509=96, srv2047121235=61, srv1159003577=5, srv402315486=73, srv678397925=87, srv139008636=15, srv408012648=74, srv714841081=88, srv1393430968=16, srv351156577=71, srv590627876=86, srv1867351047=50, srv1050833484=0, srv1583020570=30, srv1927522878=53, srv1992104131=55, srv1408434441=18, srv198380182=54, srv337231076=68, srv428614230=78, srv499812283=82, srv499442939=81, srv1107502226=2, srv1072596864=1, srv369159706=72, srv1768562234=43, srv1353268395=14, srv2008696831=58, srv1694167812=40, srv477066171=80, srv587654233=85, srv121903446=9, srv1398103414=17, srv153444522=29, srv980750520=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 17:53:29,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 17:53:29,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 17:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 17:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 17:53:29,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 17:53:29,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 17:53:29,490 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004424760000269228); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004854793875029461); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004424760000269228); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:29,490 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 17:53:29,491 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.958654225236437E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004424760000269228); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004854793875029461); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004424760000269228); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 17:53:29,843 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations. Found a solution that moves 203 regions; Going from a computed imbalance of 6.958654225236437E-4 to a new imbalance of 2.0473782677254542E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.022555555555555554); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:29,844 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1329931440:83 , srv1398103414:83 , srv153444522:83 , srv1639568859:83 , srv168032846:83 , srv294292153:83 , srv798579629:83 , srv1138817541:84 , srv1148932263:84 , srv1168949028:84 , srv121903446:84 , srv1437113798:84 , srv1441397235:84 , srv1441741628:84 , srv1478176340:84 , srv1491380947:84 , srv1502574985:84 , srv1583020570:84 , srv1612800911:84 , srv1644730159:84 , srv1678091262:84 , srv1694167812:84 , srv1728053811:84 , srv1765844386:84 , srv1788135002:84 , srv1793231635:84 , srv1814175923:84 , srv1822836179:84 , srv198380182:84 , srv1992104131:84 , srv2041799225:84 , srv2043549199:84 , srv2129463572:84 , srv262237444:84 , srv337231076:84 , srv34429903:84 , srv369159706:84 , srv408012648:84 , srv418466136:84 , srv42524708:84 , srv428614230:84 , srv477066171:84 , srv499442939:84 , srv499812283:84 , srv545851645:84 , srv587654233:84 , srv590627876:84 , srv678397925:84 , srv714841081:84 , srv862630396:84 , srv899427774:84 , srv922220509:84 , srv976750421:84 , srv980750520:84 , srv1050833484:85 , srv1072596864:85 , srv1107502226:85 , srv1159003577:85 , srv1170401864:85 , srv1202114582:85 , srv1241377500:85 , srv1253224911:85 , srv127069451:85 , srv1353268395:85 , srv139008636:85 , srv1393430968:85 , srv1408434441:85 , srv1440515616:85 , srv1444440144:85 , srv1464628303:85 , srv1467690478:85 , srv1609684384:85 , srv1671926698:85 , srv168993262:85 , srv1768562234:85 , srv1795704105:85 , srv1849693652:85 , srv1867351047:85 , srv1894120202:85 , srv1911623048:85 , srv1927522878:85 , srv1995103527:85 , srv2003549916:85 , srv2008696831:85 , srv207540720:85 , srv255730540:85 , srv266705009:85 , srv342491125:85 , srv351156577:85 , srv402315486:85 , srv417677520:85 , srv450574713:85 , srv50101477:85 , srv740983790:85 , srv750606200:85 , srv754020213:85 , srv784528201:85 , srv994644735:85 , srv1690586985:86 , srv2047121235:645 } 2023-10-24 17:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464628303=24, srv1911623048=52, srv34429903=70, srv899427774=95, srv1241377500=10, srv1788135002=44, srv1444440144=23, srv1491380947=27, srv1329931440=13, srv168993262=38, srv1822836179=48, srv42524708=77, srv50101477=83, srv750606200=90, srv1467690478=25, srv784528201=92, srv740983790=89, srv262237444=65, srv1148932263=4, srv1644730159=34, srv255730540=64, srv450574713=79, srv2041799225=59, srv1765844386=42, srv1478176340=26, srv1138817541=3, srv342491125=69, srv418466136=76, srv1814175923=47, srv1202114582=8, srv994644735=99, srv1671926698=35, srv545851645=84, srv2129463572=63, srv1678091262=36, srv168032846=37, srv976750421=97, srv1441741628=22, srv1849693652=49, srv1253224911=11, srv1168949028=6, srv1437113798=19, srv754020213=91, srv207540720=62, srv798579629=93, srv1440515616=20, srv862630396=94, srv1639568859=33, srv1728053811=41, srv294292153=67, srv1609684384=31, srv2003549916=57, srv1441397235=21, srv1995103527=56, srv2043549199=60, srv127069451=12, srv1502574985=28, srv1894120202=51, srv1690586985=39, srv1170401864=7, srv266705009=66, srv417677520=75, srv1795704105=46, srv1793231635=45, srv1612800911=32, srv922220509=96, srv2047121235=61, srv1159003577=5, srv402315486=73, srv678397925=87, srv139008636=15, srv408012648=74, srv714841081=88, srv1393430968=16, srv351156577=71, srv590627876=86, srv1867351047=50, srv1050833484=0, srv1583020570=30, srv1927522878=53, srv1992104131=55, srv1408434441=18, srv198380182=54, srv337231076=68, srv428614230=78, srv499812283=82, srv499442939=81, srv1107502226=2, srv1072596864=1, srv369159706=72, srv1768562234=43, srv1353268395=14, srv2008696831=58, srv1694167812=40, srv477066171=80, srv587654233=85, srv121903446=9, srv1398103414=17, srv153444522=29, srv980750520=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 17:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 17:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 17:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 17:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 17:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 17:53:29,974 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9052346296285582E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:29,975 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1329931440:83 , srv1398103414:83 , srv153444522:83 , srv1639568859:83 , srv168032846:83 , srv294292153:83 , srv798579629:83 , srv1138817541:84 , srv1148932263:84 , srv1168949028:84 , srv121903446:84 , srv1437113798:84 , srv1441397235:84 , srv1441741628:84 , srv1478176340:84 , srv1491380947:84 , srv1502574985:84 , srv1583020570:84 , srv1612800911:84 , srv1644730159:84 , srv1678091262:84 , srv1694167812:84 , srv1728053811:84 , srv1765844386:84 , srv1788135002:84 , srv1793231635:84 , srv1814175923:84 , srv1822836179:84 , srv198380182:84 , srv1992104131:84 , srv2041799225:84 , srv2043549199:84 , srv2129463572:84 , srv262237444:84 , srv337231076:84 , srv34429903:84 , srv369159706:84 , srv408012648:84 , srv418466136:84 , srv42524708:84 , srv428614230:84 , srv477066171:84 , srv499442939:84 , srv499812283:84 , srv545851645:84 , srv587654233:84 , srv590627876:84 , srv678397925:84 , srv714841081:84 , srv862630396:84 , srv899427774:84 , srv922220509:84 , srv976750421:84 , srv980750520:84 , srv1050833484:85 , srv1072596864:85 , srv1107502226:85 , srv1159003577:85 , srv1170401864:85 , srv1202114582:85 , srv1241377500:85 , srv1253224911:85 , srv127069451:85 , srv1353268395:85 , srv139008636:85 , srv1393430968:85 , srv1408434441:85 , srv1440515616:85 , srv1444440144:85 , srv1464628303:85 , srv1467690478:85 , srv1609684384:85 , srv1671926698:85 , srv168993262:85 , srv1768562234:85 , srv1795704105:85 , srv1849693652:85 , srv1867351047:85 , srv1894120202:85 , srv1911623048:85 , srv1927522878:85 , srv1995103527:85 , srv2003549916:85 , srv2008696831:85 , srv207540720:85 , srv255730540:85 , srv266705009:85 , srv342491125:85 , srv351156577:85 , srv402315486:85 , srv417677520:85 , srv450574713:85 , srv50101477:85 , srv740983790:85 , srv750606200:85 , srv754020213:85 , srv784528201:85 , srv994644735:85 , srv1690586985:86 , srv2047121235:645 } 2023-10-24 17:53:30,002 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 320) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7970 (was 8473) 2023-10-24 17:53:30,011 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=358, ProcessCount=169, AvailableMemoryMB=7969 2023-10-24 17:53:30,012 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-24 17:53:30,012 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-24 17:53:30,024 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005903143:84 , srv1009416302:84 , srv1113676705:84 , srv1176841804:84 , srv1189962399:84 , srv1269399091:84 , srv1309405078:84 , srv1349496965:84 , srv157634033:84 , srv1617580510:84 , srv1621759634:84 , srv1623527590:84 , srv1714203851:84 , srv1855547631:84 , srv1949276589:84 , srv1952511783:84 , srv2094095928:84 , srv379848269:84 , srv517410818:84 , srv538602102:84 , srv578978021:84 , srv599656609:84 , srv705393876:84 , srv735355928:84 , srv757337421:84 , srv838274069:84 , srv882942727:84 , srv884560304:84 , srv944639276:84 , srv1646423454:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-24 17:53:30,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:53:30,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855547631=14, srv1952511783=16, srv1617580510=9, srv1113676705=2, srv1949276589=15, srv757337421=25, srv735355928=24, srv517410818=19, srv1009416302=1, srv705393876=23, srv2094095928=17, srv578978021=21, srv944639276=29, srv1309405078=6, srv1646423454=12, srv1176841804=3, srv1269399091=5, srv1005903143=0, srv882942727=27, srv838274069=26, srv1623527590=11, srv1349496965=7, srv157634033=8, srv1714203851=13, srv1189962399=4, srv884560304=28, srv538602102=20, srv1621759634=10, srv379848269=18, srv599656609=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:53:30,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-24 17:53:30,066 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1258488431614084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:53:30,066 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912589588859739, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1258488431614084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-24 17:55:30,047 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5729415 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912589588859739 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:55:30,053 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005903143:90 , srv1009416302:90 , srv1113676705:90 , srv1176841804:90 , srv1189962399:90 , srv1269399091:90 , srv1309405078:90 , srv1349496965:90 , srv157634033:90 , srv1617580510:90 , srv1621759634:90 , srv1623527590:90 , srv1646423454:90 , srv1714203851:90 , srv1855547631:90 , srv1949276589:90 , srv1952511783:90 , srv2094095928:90 , srv379848269:90 , srv517410818:90 , srv538602102:90 , srv578978021:90 , srv599656609:90 , srv705393876:90 , srv735355928:90 , srv757337421:90 , srv838274069:90 , srv882942727:90 , srv884560304:90 , srv944639276:90 } 2023-10-24 17:55:30,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 17:55:30,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855547631=14, srv1952511783=16, srv1617580510=9, srv1113676705=2, srv1949276589=15, srv757337421=25, srv735355928=24, srv517410818=19, srv1009416302=1, srv705393876=23, srv2094095928=17, srv578978021=21, srv944639276=29, srv1309405078=6, srv1646423454=12, srv1176841804=3, srv1269399091=5, srv1005903143=0, srv882942727=27, srv838274069=26, srv1623527590=11, srv1349496965=7, srv157634033=8, srv1714203851=13, srv1189962399=4, srv884560304=28, srv538602102=20, srv1621759634=10, srv379848269=18, srv599656609=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 17:55:30,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 17:55:30,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 17:55:30,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 17:55:30,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 17:55:30,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 17:55:30,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 17:55:30,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-24 17:55:30,076 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 17:55:30,076 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1005903143:90 , srv1009416302:90 , srv1113676705:90 , srv1176841804:90 , srv1189962399:90 , srv1269399091:90 , srv1309405078:90 , srv1349496965:90 , srv157634033:90 , srv1617580510:90 , srv1621759634:90 , srv1623527590:90 , srv1646423454:90 , srv1714203851:90 , srv1855547631:90 , srv1949276589:90 , srv1952511783:90 , srv2094095928:90 , srv379848269:90 , srv517410818:90 , srv538602102:90 , srv578978021:90 , srv599656609:90 , srv705393876:90 , srv735355928:90 , srv757337421:90 , srv838274069:90 , srv882942727:90 , srv884560304:90 , srv944639276:90 } 2023-10-24 17:55:30,092 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=192 (was 358), ProcessCount=167 (was 169), AvailableMemoryMB=7907 (was 7969)