2023-06-08 17:20:53,988 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-06-08 17:20:54,229 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-06-08 17:20:54,317 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-06-08 17:20:54,367 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-08 17:20:54,373 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-08 17:20:54,386 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=95, ProcessCount=182, AvailableMemoryMB=2599 2023-06-08 17:20:54,388 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-08 17:20:54,388 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-08 17:20:54,511 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1036355150:84 , srv1055067026:84 , srv1082108654:84 , srv1136098983:84 , srv1137954668:84 , srv1153760068:84 , srv1188189034:84 , srv1196194878:84 , srv12038087:84 , srv1207920926:84 , srv1210247413:84 , srv1218018147:84 , srv1231663785:84 , srv1341566590:84 , srv1343893614:84 , srv1355954661:84 , srv1427845007:84 , srv1482132695:84 , srv1510597694:84 , srv151700257:84 , srv1532036901:84 , srv1537359306:84 , srv1582907120:84 , srv161492461:84 , srv1619513542:84 , srv1650036763:84 , srv1668328167:84 , srv1674345987:84 , srv1678422525:84 , srv1691678040:84 , srv1697904058:84 , srv1706215489:84 , srv1737991723:84 , srv1740046878:84 , srv1740827984:84 , srv174570271:84 , srv1756244401:84 , srv1776706722:84 , srv1794246311:84 , srv1824896614:84 , srv1842235054:84 , srv1855451928:84 , srv1865915315:84 , srv1884162530:84 , srv1897992695:84 , srv1904496737:84 , srv1934645237:84 , srv1966374115:84 , srv2010904221:84 , srv2022898210:84 , srv2023024965:84 , srv2036596261:84 , srv206359110:84 , srv2069823898:84 , srv2097434157:84 , srv2108831892:84 , srv2147203941:84 , srv215881323:84 , srv216228801:84 , srv243510219:84 , srv245364675:84 , srv276310712:84 , srv281327629:84 , srv285203333:84 , srv301910303:84 , srv30600245:84 , srv319512500:84 , srv322245065:84 , srv338108353:84 , srv338876175:84 , srv374953539:84 , srv384436284:84 , srv404402048:84 , srv451918451:84 , srv496600865:84 , srv506921239:84 , srv520058761:84 , srv522882392:84 , srv526831978:84 , srv532188683:84 , srv542417350:84 , srv5629350:84 , srv661200769:84 , srv664126554:84 , srv664238327:84 , srv709346909:84 , srv744434271:84 , srv753820939:84 , srv820432623:84 , srv828402710:84 , srv830797536:84 , srv884136372:84 , srv921694026:84 , srv930517367:84 , srv941298845:84 , srv968610655:84 , srv977150347:84 , srv977893056:84 , srv980739323:84 , srv1371712076:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-06-08 17:20:54,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:54,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1650036763=26, srv1855451928=42, srv496600865=75, srv1371712076=16, srv384436284=72, srv1218018147=11, srv404402048=73, srv1966374115=48, srv930517367=94, srv161492461=24, srv506921239=76, srv12038087=8, srv921694026=93, srv338876175=70, srv243510219=60, srv374953539=71, srv1934645237=47, srv532188683=80, srv2036596261=52, srv977893056=98, srv276310712=62, srv522882392=78, srv744434271=87, srv1341566590=13, srv664126554=84, srv2097434157=55, srv1537359306=22, srv1737991723=33, srv941298845=95, srv980739323=99, srv1532036901=21, srv1482132695=18, srv968610655=96, srv1188189034=6, srv1794246311=39, srv1740827984=35, srv526831978=79, srv2147203941=57, srv451918451=74, srv1055067026=1, srv1082108654=2, srv1196194878=7, srv820432623=89, srv1207920926=9, srv5629350=82, srv1210247413=10, srv322245065=68, srv1668328167=27, srv661200769=83, srv1355954661=15, srv1691678040=30, srv216228801=59, srv753820939=88, srv542417350=81, srv1824896614=40, srv1153760068=5, srv174570271=36, srv215881323=58, srv2022898210=50, srv285203333=64, srv206359110=53, srv281327629=63, srv1706215489=32, srv2069823898=54, srv338108353=69, srv1865915315=43, srv884136372=92, srv1510597694=19, srv1756244401=37, srv1136098983=3, srv2108831892=56, srv1231663785=12, srv1343893614=14, srv1036355150=0, srv30600245=66, srv1776706722=38, srv1137954668=4, srv1842235054=41, srv828402710=90, srv1884162530=44, srv245364675=61, srv301910303=65, srv1697904058=31, srv1582907120=23, srv2023024965=51, srv319512500=67, srv977150347=97, srv520058761=77, srv1740046878=34, srv151700257=20, srv1904496737=46, srv1427845007=17, srv1678422525=29, srv664238327=85, srv830797536=91, srv1619513542=25, srv1674345987=28, srv2010904221=49, srv1897992695=45, srv709346909=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:54,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:54,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:54,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:54,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:54,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:54,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:54,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:54,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:54,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:54,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:54,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:54,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:54,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:54,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:54,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:54,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:54,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:54,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:54,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:54,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:54,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:54,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:54,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:54,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:54,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:54,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:54,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:54,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:54,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:54,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:54,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 17:20:54,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 17:20:54,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 17:20:54,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 17:20:54,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 17:20:54,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 17:20:54,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 17:20:54,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 17:20:54,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 17:20:54,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 17:20:54,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 17:20:54,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 17:20:54,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 17:20:54,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 17:20:54,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 17:20:54,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 17:20:54,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 17:20:54,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 17:20:54,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 17:20:54,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 17:20:54,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 17:20:54,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 17:20:54,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 17:20:54,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 17:20:54,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 17:20:54,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 17:20:54,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 17:20:54,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 17:20:54,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 17:20:54,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 17:20:54,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 17:20:54,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 17:20:54,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 17:20:54,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 17:20:54,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 17:20:54,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 17:20:54,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 17:20:54,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 17:20:54,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 17:20:54,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 17:20:54,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 17:20:54,854 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:54,854 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 17:20:54,855 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 17:20:55,343 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 682 ms to try 5000 different iterations. Found a solution that moves 4693 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18151119821946674. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042855141610885945, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05695017216821014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5214444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042855141610885945, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17708333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:55,365 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1697904058:81 , srv30600245:81 , srv338876175:81 , srv2147203941:82 , srv980739323:82 , srv1343893614:83 , srv1582907120:83 , srv1737991723:83 , srv2023024965:83 , srv206359110:83 , srv2097434157:83 , srv520058761:83 , srv664238327:83 , srv977150347:83 , srv1207920926:84 , srv1532036901:84 , srv1650036763:84 , srv1966374115:84 , srv216228801:84 , srv276310712:84 , srv281327629:84 , srv338108353:84 , srv522882392:84 , srv542417350:84 , srv941298845:84 , srv1082108654:85 , srv1137954668:85 , srv1153760068:85 , srv1196194878:85 , srv1231663785:85 , srv1427845007:85 , srv1537359306:85 , srv1824896614:85 , srv1897992695:85 , srv1934645237:85 , srv301910303:85 , srv384436284:85 , srv451918451:85 , srv526831978:85 , srv5629350:85 , srv661200769:85 , srv709346909:85 , srv744434271:85 , srv820432623:85 , srv828402710:85 , srv930517367:85 , srv1055067026:86 , srv1341566590:86 , srv1482132695:86 , srv1510597694:86 , srv1668328167:86 , srv1740046878:86 , srv1865915315:86 , srv496600865:86 , srv921694026:86 , srv1188189034:87 , srv1210247413:87 , srv1678422525:87 , srv1740827984:87 , srv174570271:87 , srv1776706722:87 , srv1794246311:87 , srv1904496737:87 , srv285203333:87 , srv374953539:87 , srv404402048:87 , srv664126554:87 , srv753820939:87 , srv830797536:87 , srv1036355150:88 , srv12038087:88 , srv1355954661:88 , srv1674345987:88 , srv1706215489:88 , srv1756244401:88 , srv1855451928:88 , srv1884162530:88 , srv2010904221:88 , srv2022898210:88 , srv215881323:88 , srv245364675:88 , srv319512500:88 , srv884136372:88 , srv968610655:88 , srv1136098983:89 , srv1218018147:89 , srv2036596261:89 , srv2069823898:89 , srv2108831892:89 , srv322245065:89 , srv977893056:89 , srv151700257:90 , srv1619513542:90 , srv243510219:90 , srv506921239:90 , srv161492461:91 , srv1691678040:91 , srv1842235054:94 , srv532188683:95 , srv1371712076:471 } 2023-06-08 17:20:55,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:55,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1650036763=26, srv1855451928=42, srv496600865=75, srv1371712076=16, srv384436284=72, srv1218018147=11, srv404402048=73, srv1966374115=48, srv930517367=94, srv161492461=24, srv506921239=76, srv12038087=8, srv921694026=93, srv338876175=70, srv243510219=60, srv374953539=71, srv1934645237=47, srv532188683=80, srv2036596261=52, srv977893056=98, srv276310712=62, srv522882392=78, srv744434271=87, srv1341566590=13, srv664126554=84, srv2097434157=55, srv1537359306=22, srv1737991723=33, srv941298845=95, srv980739323=99, srv1532036901=21, srv1482132695=18, srv968610655=96, srv1188189034=6, srv1794246311=39, srv1740827984=35, srv526831978=79, srv2147203941=57, srv451918451=74, srv1055067026=1, srv1082108654=2, srv1196194878=7, srv820432623=89, srv1207920926=9, srv5629350=82, srv1210247413=10, srv322245065=68, srv1668328167=27, srv661200769=83, srv1355954661=15, srv1691678040=30, srv216228801=59, srv753820939=88, srv542417350=81, srv1824896614=40, srv1153760068=5, srv174570271=36, srv215881323=58, srv2022898210=50, srv285203333=64, srv206359110=53, srv281327629=63, srv1706215489=32, srv2069823898=54, srv338108353=69, srv1865915315=43, srv884136372=92, srv1510597694=19, srv1756244401=37, srv1136098983=3, srv2108831892=56, srv1231663785=12, srv1343893614=14, srv1036355150=0, srv30600245=66, srv1776706722=38, srv1137954668=4, srv1842235054=41, srv828402710=90, srv1884162530=44, srv245364675=61, srv301910303=65, srv1697904058=31, srv1582907120=23, srv2023024965=51, srv319512500=67, srv977150347=97, srv520058761=77, srv1740046878=34, srv151700257=20, srv1904496737=46, srv1427845007=17, srv1678422525=29, srv664238327=85, srv830797536=91, srv1619513542=25, srv1674345987=28, srv2010904221=49, srv1897992695=45, srv709346909=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:55,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:55,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:55,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 17:20:55,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 17:20:55,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 17:20:55,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 17:20:55,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 17:20:55,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 17:20:55,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 17:20:55,552 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042855141610885945, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05695017216821014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042855141610885945, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17708333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:55,552 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 17:20:55,552 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1814783371311126, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042855141610885945, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05695017216821014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042855141610885945, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17708333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 17:20:55,739 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 333 ms to try 5000 different iterations. Found a solution that moves 2588 regions; Going from a computed imbalance of 0.1814783371311126 to a new imbalance of 0.015057369118135986. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022396622053034183); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03873128236252405, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28755555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022396622053034183); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:55,746 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1691678040:79 , srv1842235054:79 , srv532188683:79 , srv1855451928:81 , srv1884162530:81 , srv977893056:81 , srv1036355150:82 , srv1218018147:82 , srv1355954661:82 , srv151700257:82 , srv245364675:82 , srv1136098983:83 , srv1188189034:83 , srv161492461:83 , srv1619513542:83 , srv1678422525:83 , srv1794246311:83 , srv2010904221:83 , srv2108831892:83 , srv243510219:83 , srv506921239:83 , srv664126554:83 , srv884136372:83 , srv1482132695:84 , srv1756244401:84 , srv2022898210:84 , srv2069823898:84 , srv215881323:84 , srv319512500:84 , srv374953539:84 , srv496600865:84 , srv1055067026:85 , srv1650036763:85 , srv1674345987:85 , srv1706215489:85 , srv1740046878:85 , srv1740827984:85 , srv174570271:85 , srv1776706722:85 , srv2036596261:85 , srv322245065:85 , srv709346909:85 , srv830797536:85 , srv12038087:86 , srv1210247413:86 , srv1231663785:86 , srv1537359306:86 , srv1904496737:86 , srv1934645237:86 , srv216228801:86 , srv285203333:86 , srv301910303:86 , srv404402048:86 , srv526831978:86 , srv744434271:86 , srv820432623:86 , srv828402710:86 , srv968610655:86 , srv1082108654:87 , srv1137954668:87 , srv1343893614:87 , srv1510597694:87 , srv1532036901:87 , srv1668328167:87 , srv1824896614:87 , srv281327629:87 , srv338108353:87 , srv451918451:87 , srv522882392:87 , srv5629350:87 , srv930517367:87 , srv1196194878:88 , srv1207920926:88 , srv1341566590:88 , srv1427845007:88 , srv1582907120:88 , srv1897992695:88 , srv206359110:88 , srv661200769:88 , srv753820939:88 , srv1865915315:89 , srv2023024965:89 , srv2097434157:89 , srv2147203941:89 , srv542417350:89 , srv664238327:89 , srv921694026:89 , srv941298845:89 , srv1153760068:90 , srv1966374115:90 , srv30600245:90 , srv338876175:90 , srv384436284:90 , srv980739323:90 , srv1697904058:91 , srv1737991723:91 , srv276310712:91 , srv520058761:91 , srv977150347:91 , srv1371712076:502 } 2023-06-08 17:20:55,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:55,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1650036763=26, srv1855451928=42, srv496600865=75, srv1371712076=16, srv384436284=72, srv1218018147=11, srv404402048=73, srv1966374115=48, srv930517367=94, srv161492461=24, srv506921239=76, srv12038087=8, srv921694026=93, srv338876175=70, srv243510219=60, srv374953539=71, srv1934645237=47, srv532188683=80, srv2036596261=52, srv977893056=98, srv276310712=62, srv522882392=78, srv744434271=87, srv1341566590=13, srv664126554=84, srv2097434157=55, srv1537359306=22, srv1737991723=33, srv941298845=95, srv980739323=99, srv1532036901=21, srv1482132695=18, srv968610655=96, srv1188189034=6, srv1794246311=39, srv1740827984=35, srv526831978=79, srv2147203941=57, srv451918451=74, srv1055067026=1, srv1082108654=2, srv1196194878=7, srv820432623=89, srv1207920926=9, srv5629350=82, srv1210247413=10, srv322245065=68, srv1668328167=27, srv661200769=83, srv1355954661=15, srv1691678040=30, srv216228801=59, srv753820939=88, srv542417350=81, srv1824896614=40, srv1153760068=5, srv174570271=36, srv215881323=58, srv2022898210=50, srv285203333=64, srv206359110=53, srv281327629=63, srv1706215489=32, srv2069823898=54, srv338108353=69, srv1865915315=43, srv884136372=92, srv1510597694=19, srv1756244401=37, srv1136098983=3, srv2108831892=56, srv1231663785=12, srv1343893614=14, srv1036355150=0, srv30600245=66, srv1776706722=38, srv1137954668=4, srv1842235054=41, srv828402710=90, srv1884162530=44, srv245364675=61, srv301910303=65, srv1697904058=31, srv1582907120=23, srv2023024965=51, srv319512500=67, srv977150347=97, srv520058761=77, srv1740046878=34, srv151700257=20, srv1904496737=46, srv1427845007=17, srv1678422525=29, srv664238327=85, srv830797536=91, srv1619513542=25, srv1674345987=28, srv2010904221=49, srv1897992695=45, srv709346909=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:55,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:55,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:55,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 17:20:55,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 17:20:55,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 17:20:55,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 17:20:55,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 17:20:55,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 17:20:55,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 17:20:55,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 17:20:55,859 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022396622053034183); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03873128236252405, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022396622053034183); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:55,859 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 17:20:55,860 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015039247554816048, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022396622053034183); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03873128236252405, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022396622053034183); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 17:20:56,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 216 ms to try 5000 different iterations. Found a solution that moves 979 regions; Going from a computed imbalance of 0.015039247554816048 to a new imbalance of 9.536767001811799E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010720403362764929); MoveCostFunction : (multiplier=7.0, imbalance=0.10877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,003 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1153760068:82 , srv753820939:82 , srv12038087:83 , srv1341566590:83 , srv161492461:83 , srv1737991723:83 , srv1865915315:83 , srv1966374115:83 , srv384436284:83 , srv532188683:83 , srv968610655:83 , srv1196194878:84 , srv1210247413:84 , srv1510597694:84 , srv1619513542:84 , srv1842235054:84 , srv1904496737:84 , srv2036596261:84 , srv243510219:84 , srv276310712:84 , srv322245065:84 , srv520058761:84 , srv921694026:84 , srv977150347:84 , srv1082108654:85 , srv1137954668:85 , srv1207920926:85 , srv1427845007:85 , srv151700257:85 , srv1668328167:85 , srv1674345987:85 , srv1697904058:85 , srv1706215489:85 , srv1740827984:85 , srv174570271:85 , srv1756244401:85 , srv1824896614:85 , srv1897992695:85 , srv2069823898:85 , srv2097434157:85 , srv285203333:85 , srv404402048:85 , srv451918451:85 , srv506921239:85 , srv542417350:85 , srv5629350:85 , srv661200769:85 , srv941298845:85 , srv1055067026:86 , srv1188189034:86 , srv1231663785:86 , srv1537359306:86 , srv1582907120:86 , srv1740046878:86 , srv1776706722:86 , srv1934645237:86 , srv2022898210:86 , srv2023024965:86 , srv206359110:86 , srv2108831892:86 , srv2147203941:86 , srv215881323:86 , srv281327629:86 , srv301910303:86 , srv319512500:86 , srv338108353:86 , srv338876175:86 , srv374953539:86 , srv522882392:86 , srv744434271:86 , srv830797536:86 , srv884136372:86 , srv930517367:86 , srv980739323:86 , srv1036355150:87 , srv1136098983:87 , srv1218018147:87 , srv1482132695:87 , srv1532036901:87 , srv1678422525:87 , srv1691678040:87 , srv1794246311:87 , srv2010904221:87 , srv245364675:87 , srv30600245:87 , srv526831978:87 , srv664126554:87 , srv664238327:87 , srv820432623:87 , srv828402710:87 , srv977893056:87 , srv1343893614:88 , srv1355954661:88 , srv1650036763:88 , srv1855451928:88 , srv1884162530:88 , srv216228801:88 , srv709346909:88 , srv496600865:89 , srv1371712076:537 } 2023-06-08 17:20:56,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:56,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1650036763=26, srv1855451928=42, srv496600865=75, srv1371712076=16, srv384436284=72, srv1218018147=11, srv404402048=73, srv1966374115=48, srv930517367=94, srv161492461=24, srv506921239=76, srv12038087=8, srv921694026=93, srv338876175=70, srv243510219=60, srv374953539=71, srv1934645237=47, srv532188683=80, srv2036596261=52, srv977893056=98, srv276310712=62, srv522882392=78, srv744434271=87, srv1341566590=13, srv664126554=84, srv2097434157=55, srv1537359306=22, srv1737991723=33, srv941298845=95, srv980739323=99, srv1532036901=21, srv1482132695=18, srv968610655=96, srv1188189034=6, srv1794246311=39, srv1740827984=35, srv526831978=79, srv2147203941=57, srv451918451=74, srv1055067026=1, srv1082108654=2, srv1196194878=7, srv820432623=89, srv1207920926=9, srv5629350=82, srv1210247413=10, srv322245065=68, srv1668328167=27, srv661200769=83, srv1355954661=15, srv1691678040=30, srv216228801=59, srv753820939=88, srv542417350=81, srv1824896614=40, srv1153760068=5, srv174570271=36, srv215881323=58, srv2022898210=50, srv285203333=64, srv206359110=53, srv281327629=63, srv1706215489=32, srv2069823898=54, srv338108353=69, srv1865915315=43, srv884136372=92, srv1510597694=19, srv1756244401=37, srv1136098983=3, srv2108831892=56, srv1231663785=12, srv1343893614=14, srv1036355150=0, srv30600245=66, srv1776706722=38, srv1137954668=4, srv1842235054=41, srv828402710=90, srv1884162530=44, srv245364675=61, srv301910303=65, srv1697904058=31, srv1582907120=23, srv2023024965=51, srv319512500=67, srv977150347=97, srv520058761=77, srv1740046878=34, srv151700257=20, srv1904496737=46, srv1427845007=17, srv1678422525=29, srv664238327=85, srv830797536=91, srv1619513542=25, srv1674345987=28, srv2010904221=49, srv1897992695=45, srv709346909=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:56,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 17:20:56,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 17:20:56,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 17:20:56,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 17:20:56,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 17:20:56,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 17:20:56,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 17:20:56,099 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010720403362764929); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,099 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 17:20:56,099 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.468215956640933E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010720403362764929); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 17:20:56,299 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 270 ms to try 5000 different iterations. Found a solution that moves 239 regions; Going from a computed imbalance of 9.468215956640933E-4 to a new imbalance of 5.024563708143361E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.026555555555555554, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,300 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1136098983:83 , srv1355954661:83 , srv2036596261:83 , srv276310712:83 , srv496600865:83 , srv664238327:83 , srv709346909:83 , srv921694026:83 , srv1218018147:84 , srv1343893614:84 , srv1532036901:84 , srv1650036763:84 , srv1674345987:84 , srv1706215489:84 , srv1776706722:84 , srv1824896614:84 , srv1865915315:84 , srv1897992695:84 , srv2010904221:84 , srv2022898210:84 , srv2023024965:84 , srv2069823898:84 , srv2108831892:84 , srv216228801:84 , srv285203333:84 , srv30600245:84 , srv319512500:84 , srv322245065:84 , srv384436284:84 , srv404402048:84 , srv506921239:84 , srv520058761:84 , srv526831978:84 , srv542417350:84 , srv661200769:84 , srv753820939:84 , srv820432623:84 , srv828402710:84 , srv830797536:84 , srv930517367:84 , srv941298845:84 , srv968610655:84 , srv977150347:84 , srv980739323:84 , srv1036355150:85 , srv1055067026:85 , srv1082108654:85 , srv1137954668:85 , srv1153760068:85 , srv1196194878:85 , srv12038087:85 , srv1207920926:85 , srv1210247413:85 , srv1231663785:85 , srv1427845007:85 , srv1482132695:85 , srv1510597694:85 , srv151700257:85 , srv1537359306:85 , srv1582907120:85 , srv161492461:85 , srv1619513542:85 , srv1668328167:85 , srv1678422525:85 , srv1691678040:85 , srv1697904058:85 , srv1737991723:85 , srv1740046878:85 , srv1740827984:85 , srv174570271:85 , srv1756244401:85 , srv1794246311:85 , srv1842235054:85 , srv1855451928:85 , srv1884162530:85 , srv1904496737:85 , srv1934645237:85 , srv1966374115:85 , srv2097434157:85 , srv2147203941:85 , srv215881323:85 , srv243510219:85 , srv245364675:85 , srv281327629:85 , srv338108353:85 , srv338876175:85 , srv451918451:85 , srv522882392:85 , srv532188683:85 , srv5629350:85 , srv664126554:85 , srv744434271:85 , srv884136372:85 , srv977893056:85 , srv1188189034:86 , srv1341566590:86 , srv206359110:86 , srv301910303:86 , srv374953539:86 , srv1371712076:632 } 2023-06-08 17:20:56,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:56,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1650036763=26, srv1855451928=42, srv496600865=75, srv1371712076=16, srv384436284=72, srv1218018147=11, srv404402048=73, srv1966374115=48, srv930517367=94, srv161492461=24, srv506921239=76, srv12038087=8, srv921694026=93, srv338876175=70, srv243510219=60, srv374953539=71, srv1934645237=47, srv532188683=80, srv2036596261=52, srv977893056=98, srv276310712=62, srv522882392=78, srv744434271=87, srv1341566590=13, srv664126554=84, srv2097434157=55, srv1537359306=22, srv1737991723=33, srv941298845=95, srv980739323=99, srv1532036901=21, srv1482132695=18, srv968610655=96, srv1188189034=6, srv1794246311=39, srv1740827984=35, srv526831978=79, srv2147203941=57, srv451918451=74, srv1055067026=1, srv1082108654=2, srv1196194878=7, srv820432623=89, srv1207920926=9, srv5629350=82, srv1210247413=10, srv322245065=68, srv1668328167=27, srv661200769=83, srv1355954661=15, srv1691678040=30, srv216228801=59, srv753820939=88, srv542417350=81, srv1824896614=40, srv1153760068=5, srv174570271=36, srv215881323=58, srv2022898210=50, srv285203333=64, srv206359110=53, srv281327629=63, srv1706215489=32, srv2069823898=54, srv338108353=69, srv1865915315=43, srv884136372=92, srv1510597694=19, srv1756244401=37, srv1136098983=3, srv2108831892=56, srv1231663785=12, srv1343893614=14, srv1036355150=0, srv30600245=66, srv1776706722=38, srv1137954668=4, srv1842235054=41, srv828402710=90, srv1884162530=44, srv245364675=61, srv301910303=65, srv1697904058=31, srv1582907120=23, srv2023024965=51, srv319512500=67, srv977150347=97, srv520058761=77, srv1740046878=34, srv151700257=20, srv1904496737=46, srv1427845007=17, srv1678422525=29, srv664238327=85, srv830797536=91, srv1619513542=25, srv1674345987=28, srv2010904221=49, srv1897992695=45, srv709346909=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:56,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:56,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:56,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:56,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:56,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:56,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:56,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 17:20:56,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 17:20:56,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 17:20:56,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 17:20:56,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 17:20:56,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 17:20:56,395 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,395 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 17:20:56,396 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.857212331270661E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 17:20:56,520 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 195 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 4.857212331270661E-5 to a new imbalance of 7.604824139770872E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,520 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1341566590:83 , srv1427845007:83 , srv1668328167:83 , srv206359110:83 , srv215881323:83 , srv301910303:83 , srv374953539:83 , srv1036355150:84 , srv1055067026:84 , srv1082108654:84 , srv1136098983:84 , srv1137954668:84 , srv1153760068:84 , srv1188189034:84 , srv1196194878:84 , srv12038087:84 , srv1207920926:84 , srv1210247413:84 , srv1218018147:84 , srv1231663785:84 , srv1343893614:84 , srv1482132695:84 , srv1510597694:84 , srv151700257:84 , srv1532036901:84 , srv1537359306:84 , srv1582907120:84 , srv161492461:84 , srv1619513542:84 , srv1674345987:84 , srv1678422525:84 , srv1691678040:84 , srv1697904058:84 , srv1706215489:84 , srv1737991723:84 , srv1740046878:84 , srv1740827984:84 , srv174570271:84 , srv1756244401:84 , srv1776706722:84 , srv1794246311:84 , srv1842235054:84 , srv1855451928:84 , srv1865915315:84 , srv1884162530:84 , srv1897992695:84 , srv1904496737:84 , srv1934645237:84 , srv1966374115:84 , srv2010904221:84 , srv2022898210:84 , srv2023024965:84 , srv2069823898:84 , srv2097434157:84 , srv2108831892:84 , srv2147203941:84 , srv216228801:84 , srv243510219:84 , srv245364675:84 , srv276310712:84 , srv281327629:84 , srv285203333:84 , srv30600245:84 , srv319512500:84 , srv322245065:84 , srv338108353:84 , srv338876175:84 , srv384436284:84 , srv404402048:84 , srv451918451:84 , srv496600865:84 , srv506921239:84 , srv520058761:84 , srv522882392:84 , srv526831978:84 , srv532188683:84 , srv542417350:84 , srv5629350:84 , srv661200769:84 , srv664126554:84 , srv664238327:84 , srv744434271:84 , srv820432623:84 , srv828402710:84 , srv830797536:84 , srv884136372:84 , srv921694026:84 , srv930517367:84 , srv941298845:84 , srv977150347:84 , srv977893056:84 , srv980739323:84 , srv1355954661:85 , srv1650036763:85 , srv1824896614:85 , srv2036596261:85 , srv709346909:85 , srv753820939:85 , srv968610655:85 , srv1371712076:684 } 2023-06-08 17:20:56,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:56,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1650036763=26, srv1855451928=42, srv496600865=75, srv1371712076=16, srv384436284=72, srv1218018147=11, srv404402048=73, srv1966374115=48, srv930517367=94, srv161492461=24, srv506921239=76, srv12038087=8, srv921694026=93, srv338876175=70, srv243510219=60, srv374953539=71, srv1934645237=47, srv532188683=80, srv2036596261=52, srv977893056=98, srv276310712=62, srv522882392=78, srv744434271=87, srv1341566590=13, srv664126554=84, srv2097434157=55, srv1537359306=22, srv1737991723=33, srv941298845=95, srv980739323=99, srv1532036901=21, srv1482132695=18, srv968610655=96, srv1188189034=6, srv1794246311=39, srv1740827984=35, srv526831978=79, srv2147203941=57, srv451918451=74, srv1055067026=1, srv1082108654=2, srv1196194878=7, srv820432623=89, srv1207920926=9, srv5629350=82, srv1210247413=10, srv322245065=68, srv1668328167=27, srv661200769=83, srv1355954661=15, srv1691678040=30, srv216228801=59, srv753820939=88, srv542417350=81, srv1824896614=40, srv1153760068=5, srv174570271=36, srv215881323=58, srv2022898210=50, srv285203333=64, srv206359110=53, srv281327629=63, srv1706215489=32, srv2069823898=54, srv338108353=69, srv1865915315=43, srv884136372=92, srv1510597694=19, srv1756244401=37, srv1136098983=3, srv2108831892=56, srv1231663785=12, srv1343893614=14, srv1036355150=0, srv30600245=66, srv1776706722=38, srv1137954668=4, srv1842235054=41, srv828402710=90, srv1884162530=44, srv245364675=61, srv301910303=65, srv1697904058=31, srv1582907120=23, srv2023024965=51, srv319512500=67, srv977150347=97, srv520058761=77, srv1740046878=34, srv151700257=20, srv1904496737=46, srv1427845007=17, srv1678422525=29, srv664238327=85, srv830797536=91, srv1619513542=25, srv1674345987=28, srv2010904221=49, srv1897992695=45, srv709346909=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:56,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:56,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 17:20:56,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 17:20:56,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 17:20:56,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 17:20:56,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 17:20:56,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 17:20:56,615 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,615 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1341566590:83 , srv1427845007:83 , srv1668328167:83 , srv206359110:83 , srv215881323:83 , srv301910303:83 , srv374953539:83 , srv1036355150:84 , srv1055067026:84 , srv1082108654:84 , srv1136098983:84 , srv1137954668:84 , srv1153760068:84 , srv1188189034:84 , srv1196194878:84 , srv12038087:84 , srv1207920926:84 , srv1210247413:84 , srv1218018147:84 , srv1231663785:84 , srv1343893614:84 , srv1482132695:84 , srv1510597694:84 , srv151700257:84 , srv1532036901:84 , srv1537359306:84 , srv1582907120:84 , srv161492461:84 , srv1619513542:84 , srv1674345987:84 , srv1678422525:84 , srv1691678040:84 , srv1697904058:84 , srv1706215489:84 , srv1737991723:84 , srv1740046878:84 , srv1740827984:84 , srv174570271:84 , srv1756244401:84 , srv1776706722:84 , srv1794246311:84 , srv1842235054:84 , srv1855451928:84 , srv1865915315:84 , srv1884162530:84 , srv1897992695:84 , srv1904496737:84 , srv1934645237:84 , srv1966374115:84 , srv2010904221:84 , srv2022898210:84 , srv2023024965:84 , srv2069823898:84 , srv2097434157:84 , srv2108831892:84 , srv2147203941:84 , srv216228801:84 , srv243510219:84 , srv245364675:84 , srv276310712:84 , srv281327629:84 , srv285203333:84 , srv30600245:84 , srv319512500:84 , srv322245065:84 , srv338108353:84 , srv338876175:84 , srv384436284:84 , srv404402048:84 , srv451918451:84 , srv496600865:84 , srv506921239:84 , srv520058761:84 , srv522882392:84 , srv526831978:84 , srv532188683:84 , srv542417350:84 , srv5629350:84 , srv661200769:84 , srv664126554:84 , srv664238327:84 , srv744434271:84 , srv820432623:84 , srv828402710:84 , srv830797536:84 , srv884136372:84 , srv921694026:84 , srv930517367:84 , srv941298845:84 , srv977150347:84 , srv977893056:84 , srv980739323:84 , srv1355954661:85 , srv1650036763:85 , srv1824896614:85 , srv2036596261:85 , srv709346909:85 , srv753820939:85 , srv968610655:85 , srv1371712076:684 } 2023-06-08 17:20:56,636 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=112 (was 95) - SystemLoadAverage LEAK? -, ProcessCount=182 (was 182), AvailableMemoryMB=2112 (was 2599) 2023-06-08 17:20:56,645 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=112, ProcessCount=182, AvailableMemoryMB=2111 2023-06-08 17:20:56,645 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-08 17:20:56,646 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-06-08 17:20:56,653 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037196247:84 , srv1092989293:84 , srv1135111240:84 , srv1344946709:84 , srv1353585089:84 , srv1463542416:84 , srv1519209798:84 , srv1552182236:84 , srv1581009533:84 , srv1656628124:84 , srv1669950009:84 , srv1947744256:84 , srv1950700261:84 , srv197192166:84 , srv2081672897:84 , srv264844378:84 , srv28659710:84 , srv342326398:84 , srv344411505:84 , srv353592745:84 , srv423943468:84 , srv558295532:84 , srv644439858:84 , srv742127057:84 , srv765271435:84 , srv811047935:84 , srv814356873:84 , srv882822049:84 , srv894320850:84 , srv1495515552:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-06-08 17:20:56,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:20:56,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv882822049=28, srv1135111240=2, srv894320850=29, srv1463542416=5, srv1581009533=9, srv28659710=17, srv811047935=26, srv264844378=16, srv1037196247=0, srv1092989293=1, srv197192166=14, srv1353585089=4, srv1950700261=13, srv342326398=18, srv765271435=25, srv814356873=27, srv344411505=19, srv742127057=24, srv423943468=21, srv1669950009=11, srv353592745=20, srv1947744256=12, srv1495515552=6, srv1344946709=3, srv558295532=22, srv2081672897=15, srv644439858=23, srv1519209798=7, srv1552182236=8, srv1656628124=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:20:56,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:20:56,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:20:56,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:20:56,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:20:56,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-08 17:20:56,681 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1460140429121164, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:20:56,681 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912653128745699, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1460140429121164, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-06-08 17:22:56,670 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 11011395 different iterations. Found a solution that moves 1835 regions; Going from a computed imbalance of 0.9912653128745699 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:22:56,676 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037196247:90 , srv1092989293:90 , srv1135111240:90 , srv1344946709:90 , srv1353585089:90 , srv1463542416:90 , srv1495515552:90 , srv1519209798:90 , srv1552182236:90 , srv1581009533:90 , srv1656628124:90 , srv1669950009:90 , srv1947744256:90 , srv1950700261:90 , srv197192166:90 , srv2081672897:90 , srv264844378:90 , srv28659710:90 , srv342326398:90 , srv344411505:90 , srv353592745:90 , srv423943468:90 , srv558295532:90 , srv644439858:90 , srv742127057:90 , srv765271435:90 , srv811047935:90 , srv814356873:90 , srv882822049:90 , srv894320850:90 } 2023-06-08 17:22:56,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 17:22:56,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv882822049=28, srv1135111240=2, srv894320850=29, srv1463542416=5, srv1581009533=9, srv28659710=17, srv811047935=26, srv264844378=16, srv1037196247=0, srv1092989293=1, srv197192166=14, srv1353585089=4, srv1950700261=13, srv342326398=18, srv765271435=25, srv814356873=27, srv344411505=19, srv742127057=24, srv423943468=21, srv1669950009=11, srv353592745=20, srv1947744256=12, srv1495515552=6, srv1344946709=3, srv558295532=22, srv2081672897=15, srv644439858=23, srv1519209798=7, srv1552182236=8, srv1656628124=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 17:22:56,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 17:22:56,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 17:22:56,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 17:22:56,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 17:22:56,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-08 17:22:56,691 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 17:22:56,692 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1037196247:90 , srv1092989293:90 , srv1135111240:90 , srv1344946709:90 , srv1353585089:90 , srv1463542416:90 , srv1495515552:90 , srv1519209798:90 , srv1552182236:90 , srv1581009533:90 , srv1656628124:90 , srv1669950009:90 , srv1947744256:90 , srv1950700261:90 , srv197192166:90 , srv2081672897:90 , srv264844378:90 , srv28659710:90 , srv342326398:90 , srv344411505:90 , srv353592745:90 , srv423943468:90 , srv558295532:90 , srv644439858:90 , srv742127057:90 , srv765271435:90 , srv811047935:90 , srv814356873:90 , srv882822049:90 , srv894320850:90 } 2023-06-08 17:22:56,705 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=118 (was 112) - SystemLoadAverage LEAK? -, ProcessCount=189 (was 182) - ProcessCount LEAK? -, AvailableMemoryMB=1886 (was 2111)