2023-10-21 23:53:27,771 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-21 23:53:28,166 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-21 23:53:28,285 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-21 23:53:28,349 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-21 23:53:28,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-21 23:53:28,368 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=171, AvailableMemoryMB=8695 2023-10-21 23:53:28,370 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-21 23:53:28,370 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-21 23:53:28,488 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019185234:84 , srv1049728892:84 , srv1073415045:84 , srv1085306726:84 , srv1095557362:84 , srv1096612029:84 , srv1120430302:84 , srv113810627:84 , srv1220181173:84 , srv1229662897:84 , srv1246567800:84 , srv1274638483:84 , srv1284107032:84 , srv1290924388:84 , srv1305873640:84 , srv1309990787:84 , srv1322589169:84 , srv1326286179:84 , srv1417244692:84 , srv142389443:84 , srv145339652:84 , srv1453843232:84 , srv147741240:84 , srv1517615689:84 , srv1533237564:84 , srv158974173:84 , srv1602079367:84 , srv1639527342:84 , srv1655297088:84 , srv1666472927:84 , srv1670556048:84 , srv1688134062:84 , srv1698101199:84 , srv1735410586:84 , srv175258404:84 , srv176547975:84 , srv1766169289:84 , srv1790185385:84 , srv1800606036:84 , srv1823518301:84 , srv1868591960:84 , srv1889694851:84 , srv190063787:84 , srv1903933800:84 , srv1915954917:84 , srv1930341904:84 , srv1937591761:84 , srv1966705335:84 , srv1972075018:84 , srv1977912307:84 , srv1989299476:84 , srv2007893908:84 , srv2014465970:84 , srv2015632305:84 , srv2066275064:84 , srv2091462929:84 , srv2099748639:84 , srv2120521873:84 , srv213003577:84 , srv234283435:84 , srv266101260:84 , srv300753910:84 , srv30186204:84 , srv331736176:84 , srv412130545:84 , srv453041801:84 , srv457089955:84 , srv469062581:84 , srv485930826:84 , srv502888393:84 , srv519445519:84 , srv545675838:84 , srv546890675:84 , srv54995219:84 , srv551407204:84 , srv555982848:84 , srv586356351:84 , srv587985337:84 , srv624626249:84 , srv631059494:84 , srv6384212:84 , srv679515436:84 , srv687535454:84 , srv705603278:84 , srv723936452:84 , srv743137411:84 , srv745625445:84 , srv75901707:84 , srv769249723:84 , srv769610212:84 , srv805630119:84 , srv831407119:84 , srv83432528:84 , srv864550131:84 , srv901477338:84 , srv953988696:84 , srv960038509:84 , srv978064746:84 , srv994592199:84 , srv1811556060:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-21 23:53:28,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv469062581=68, srv1220181173=8, srv1309990787=15, srv687535454=83, srv213003577=59, srv1120430302=6, srv1533237564=24, srv994592199=99, srv2099748639=57, srv453041801=66, srv1800606036=38, srv1811556060=39, srv1096612029=5, srv412130545=65, srv1666472927=29, srv1095557362=4, srv831407119=92, srv1453843232=21, srv54995219=74, srv587985337=78, srv631059494=80, srv1322589169=16, srv1670556048=30, srv864550131=94, srv30186204=63, srv519445519=71, srv545675838=72, srv1229662897=9, srv158974173=25, srv145339652=20, srv901477338=95, srv1889694851=42, srv266101260=61, srv2091462929=56, srv142389443=19, srv331736176=64, srv1937591761=47, srv1989299476=51, srv745625445=87, srv769610212=90, srv705603278=84, srv2014465970=53, srv502888393=70, srv743137411=86, srv1417244692=18, srv624626249=79, srv1049728892=1, srv1766169289=36, srv234283435=60, srv1073415045=2, srv2015632305=54, srv2066275064=55, srv1246567800=10, srv1517615689=23, srv190063787=43, srv551407204=75, srv147741240=22, srv1972075018=49, srv1019185234=0, srv1903933800=44, srv586356351=77, srv679515436=82, srv960038509=97, srv457089955=67, srv300753910=62, srv1305873640=14, srv75901707=88, srv723936452=85, srv1698101199=32, srv2007893908=52, srv1930341904=46, srv1274638483=11, srv1966705335=48, srv1915954917=45, srv1688134062=31, srv485930826=69, srv1290924388=13, srv175258404=34, srv1085306726=3, srv1284107032=12, srv555982848=76, srv546890675=73, srv83432528=93, srv1790185385=37, srv1823518301=40, srv1977912307=50, srv1735410586=33, srv953988696=96, srv2120521873=58, srv1868591960=41, srv1602079367=26, srv1326286179=17, srv6384212=81, srv113810627=7, srv176547975=35, srv805630119=91, srv1655297088=28, srv1639527342=27, srv769249723=89, srv978064746=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-21 23:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-21 23:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-21 23:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-21 23:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-21 23:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-21 23:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-21 23:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-21 23:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-21 23:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-21 23:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-21 23:53:28,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:28,973 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-21 23:53:28,973 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-21 23:53:29,397 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 747 ms to try 5000 different iterations. Found a solution that moves 4705 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.180690019269926. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0432239835878375, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05577532289659987, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5227777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0432239835878375, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24408333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:29,422 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1290924388:78 , srv2007893908:79 , srv586356351:79 , srv587985337:79 , srv1735410586:81 , srv1800606036:82 , srv2091462929:82 , srv1096612029:83 , srv176547975:83 , srv457089955:83 , srv485930826:83 , srv502888393:83 , srv545675838:83 , srv546890675:83 , srv953988696:83 , srv1229662897:84 , srv1246567800:84 , srv1309990787:84 , srv1453843232:84 , srv1639527342:84 , srv1915954917:84 , srv1937591761:84 , srv266101260:84 , srv469062581:84 , srv705603278:84 , srv723936452:84 , srv743137411:84 , srv745625445:84 , srv769610212:84 , srv864550131:84 , srv901477338:84 , srv1095557362:85 , srv147741240:85 , srv1666472927:85 , srv2015632305:85 , srv2099748639:85 , srv2120521873:85 , srv213003577:85 , srv631059494:85 , srv6384212:85 , srv83432528:85 , srv1019185234:86 , srv1085306726:86 , srv1120430302:86 , srv1274638483:86 , srv1305873640:86 , srv142389443:86 , srv158974173:86 , srv1790185385:86 , srv1930341904:86 , srv2014465970:86 , srv30186204:86 , srv453041801:86 , srv519445519:86 , srv551407204:86 , srv1049728892:87 , srv1326286179:87 , srv145339652:87 , srv1533237564:87 , srv1655297088:87 , srv175258404:87 , srv1977912307:87 , srv412130545:87 , srv54995219:87 , srv555982848:87 , srv831407119:87 , srv1220181173:88 , srv1284107032:88 , srv1322589169:88 , srv1417244692:88 , srv1517615689:88 , srv1670556048:88 , srv1766169289:88 , srv1889694851:88 , srv190063787:88 , srv234283435:88 , srv300753910:88 , srv331736176:88 , srv687535454:88 , srv1602079367:89 , srv1698101199:89 , srv1823518301:89 , srv1868591960:89 , srv1966705335:89 , srv1989299476:89 , srv679515436:89 , srv75901707:89 , srv769249723:89 , srv960038509:89 , srv978064746:89 , srv113810627:90 , srv1903933800:90 , srv2066275064:90 , srv1688134062:91 , srv624626249:91 , srv1972075018:92 , srv805630119:92 , srv1073415045:93 , srv994592199:95 , srv1811556060:474 } 2023-10-21 23:53:29,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv469062581=68, srv1220181173=8, srv1309990787=15, srv687535454=83, srv213003577=59, srv1120430302=6, srv1533237564=24, srv994592199=99, srv2099748639=57, srv453041801=66, srv1800606036=38, srv1811556060=39, srv1096612029=5, srv412130545=65, srv1666472927=29, srv1095557362=4, srv831407119=92, srv1453843232=21, srv54995219=74, srv587985337=78, srv631059494=80, srv1322589169=16, srv1670556048=30, srv864550131=94, srv30186204=63, srv519445519=71, srv545675838=72, srv1229662897=9, srv158974173=25, srv145339652=20, srv901477338=95, srv1889694851=42, srv266101260=61, srv2091462929=56, srv142389443=19, srv331736176=64, srv1937591761=47, srv1989299476=51, srv745625445=87, srv769610212=90, srv705603278=84, srv2014465970=53, srv502888393=70, srv743137411=86, srv1417244692=18, srv624626249=79, srv1049728892=1, srv1766169289=36, srv234283435=60, srv1073415045=2, srv2015632305=54, srv2066275064=55, srv1246567800=10, srv1517615689=23, srv190063787=43, srv551407204=75, srv147741240=22, srv1972075018=49, srv1019185234=0, srv1903933800=44, srv586356351=77, srv679515436=82, srv960038509=97, srv457089955=67, srv300753910=62, srv1305873640=14, srv75901707=88, srv723936452=85, srv1698101199=32, srv2007893908=52, srv1930341904=46, srv1274638483=11, srv1966705335=48, srv1915954917=45, srv1688134062=31, srv485930826=69, srv1290924388=13, srv175258404=34, srv1085306726=3, srv1284107032=12, srv555982848=76, srv546890675=73, srv83432528=93, srv1790185385=37, srv1823518301=40, srv1977912307=50, srv1735410586=33, srv953988696=96, srv2120521873=58, srv1868591960=41, srv1602079367=26, srv1326286179=17, srv6384212=81, srv113810627=7, srv176547975=35, srv805630119=91, srv1655297088=28, srv1639527342=27, srv769249723=89, srv978064746=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:29,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:29,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:29,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:29,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:29,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:29,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:29,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-21 23:53:29,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-21 23:53:29,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-21 23:53:29,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-21 23:53:29,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-21 23:53:29,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-21 23:53:29,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-21 23:53:29,631 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0432239835878375, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05577532289659987, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0432239835878375, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24408333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:29,631 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-21 23:53:29,631 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18065707415577595, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0432239835878375, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05577532289659987, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0432239835878375, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24408333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-21 23:53:29,902 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 435 ms to try 5000 different iterations. Found a solution that moves 2631 regions; Going from a computed imbalance of 0.18065707415577595 to a new imbalance of 0.014896358242996724. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024088314876309774); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024088314876309774); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:29,910 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv994592199:75 , srv1073415045:76 , srv805630119:79 , srv113810627:80 , srv1688134062:80 , srv1823518301:80 , srv1049728892:81 , srv1972075018:81 , srv1989299476:81 , srv1966705335:82 , srv2066275064:82 , srv624626249:82 , srv75901707:82 , srv1284107032:83 , srv1517615689:83 , srv1670556048:83 , srv1790185385:83 , srv190063787:83 , srv234283435:83 , srv331736176:83 , srv412130545:83 , srv960038509:83 , srv978064746:83 , srv1322589169:84 , srv142389443:84 , srv1602079367:84 , srv1655297088:84 , srv1698101199:84 , srv1868591960:84 , srv1889694851:84 , srv1903933800:84 , srv1930341904:84 , srv2099748639:84 , srv30186204:84 , srv769249723:84 , srv1085306726:85 , srv1220181173:85 , srv1417244692:85 , srv145339652:85 , srv1766169289:85 , srv1977912307:85 , srv300753910:85 , srv54995219:85 , srv555982848:85 , srv83432528:85 , srv1095557362:86 , srv1246567800:86 , srv1305873640:86 , srv1309990787:86 , srv1326286179:86 , srv1533237564:86 , srv1639527342:86 , srv175258404:86 , srv1915954917:86 , srv469062581:86 , srv502888393:86 , srv519445519:86 , srv6384212:86 , srv679515436:86 , srv705603278:86 , srv831407119:86 , srv901477338:86 , srv1019185234:87 , srv1229662897:87 , srv1274638483:87 , srv1453843232:87 , srv1666472927:87 , srv1937591761:87 , srv546890675:87 , srv631059494:87 , srv687535454:87 , srv769610212:87 , srv1096612029:88 , srv1120430302:88 , srv158974173:88 , srv176547975:88 , srv213003577:88 , srv453041801:88 , srv743137411:88 , srv745625445:88 , srv1800606036:89 , srv2014465970:89 , srv2015632305:89 , srv2091462929:89 , srv457089955:89 , srv551407204:89 , srv723936452:89 , srv864550131:89 , srv147741240:90 , srv1735410586:90 , srv2120521873:90 , srv485930826:90 , srv545675838:90 , srv953988696:91 , srv266101260:92 , srv586356351:92 , srv587985337:93 , srv1290924388:95 , srv2007893908:96 , srv1811556060:514 } 2023-10-21 23:53:29,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:29,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv469062581=68, srv1220181173=8, srv1309990787=15, srv687535454=83, srv213003577=59, srv1120430302=6, srv1533237564=24, srv994592199=99, srv2099748639=57, srv453041801=66, srv1800606036=38, srv1811556060=39, srv1096612029=5, srv412130545=65, srv1666472927=29, srv1095557362=4, srv831407119=92, srv1453843232=21, srv54995219=74, srv587985337=78, srv631059494=80, srv1322589169=16, srv1670556048=30, srv864550131=94, srv30186204=63, srv519445519=71, srv545675838=72, srv1229662897=9, srv158974173=25, srv145339652=20, srv901477338=95, srv1889694851=42, srv266101260=61, srv2091462929=56, srv142389443=19, srv331736176=64, srv1937591761=47, srv1989299476=51, srv745625445=87, srv769610212=90, srv705603278=84, srv2014465970=53, srv502888393=70, srv743137411=86, srv1417244692=18, srv624626249=79, srv1049728892=1, srv1766169289=36, srv234283435=60, srv1073415045=2, srv2015632305=54, srv2066275064=55, srv1246567800=10, srv1517615689=23, srv190063787=43, srv551407204=75, srv147741240=22, srv1972075018=49, srv1019185234=0, srv1903933800=44, srv586356351=77, srv679515436=82, srv960038509=97, srv457089955=67, srv300753910=62, srv1305873640=14, srv75901707=88, srv723936452=85, srv1698101199=32, srv2007893908=52, srv1930341904=46, srv1274638483=11, srv1966705335=48, srv1915954917=45, srv1688134062=31, srv485930826=69, srv1290924388=13, srv175258404=34, srv1085306726=3, srv1284107032=12, srv555982848=76, srv546890675=73, srv83432528=93, srv1790185385=37, srv1823518301=40, srv1977912307=50, srv1735410586=33, srv953988696=96, srv2120521873=58, srv1868591960=41, srv1602079367=26, srv1326286179=17, srv6384212=81, srv113810627=7, srv176547975=35, srv805630119=91, srv1655297088=28, srv1639527342=27, srv769249723=89, srv978064746=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:29,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-21 23:53:29,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-21 23:53:29,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-21 23:53:29,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-21 23:53:29,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-21 23:53:29,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-21 23:53:29,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-21 23:53:29,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-21 23:53:30,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024088314876309774); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024088314876309774); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:30,131 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-21 23:53:30,131 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014877935587241407, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024088314876309774); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024088314876309774); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-21 23:53:30,435 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 468 ms to try 5000 different iterations. Found a solution that moves 1034 regions; Going from a computed imbalance of 0.014877935587241407 to a new imbalance of 7.661961157759938E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006547761159855293); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006598983018610534); MoveCostFunction : (multiplier=7.0, imbalance=0.11488888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006547761159855293); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:30,438 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv266101260:81 , srv2014465970:82 , srv1120430302:83 , srv1290924388:83 , srv147741240:83 , srv158974173:83 , srv1903933800:83 , srv2007893908:83 , srv2015632305:83 , srv2120521873:83 , srv551407204:83 , srv679515436:83 , srv687535454:83 , srv953988696:83 , srv1019185234:84 , srv1326286179:84 , srv1417244692:84 , srv1602079367:84 , srv1868591960:84 , srv1972075018:84 , srv2066275064:84 , srv453041801:84 , srv545675838:84 , srv624626249:84 , srv831407119:84 , srv1220181173:85 , srv1274638483:85 , srv1284107032:85 , srv1305873640:85 , srv1322589169:85 , srv145339652:85 , srv1666472927:85 , srv1698101199:85 , srv175258404:85 , srv1766169289:85 , srv213003577:85 , srv300753910:85 , srv457089955:85 , srv485930826:85 , srv519445519:85 , srv555982848:85 , srv586356351:85 , srv587985337:85 , srv631059494:85 , srv723936452:85 , srv743137411:85 , srv745625445:85 , srv769249723:85 , srv1229662897:86 , srv1453843232:86 , srv1533237564:86 , srv1655297088:86 , srv1735410586:86 , srv176547975:86 , srv1889694851:86 , srv190063787:86 , srv1966705335:86 , srv1977912307:86 , srv2091462929:86 , srv234283435:86 , srv331736176:86 , srv54995219:86 , srv6384212:86 , srv75901707:86 , srv805630119:86 , srv864550131:86 , srv978064746:86 , srv1085306726:87 , srv1095557362:87 , srv1096612029:87 , srv113810627:87 , srv1309990787:87 , srv1517615689:87 , srv1639527342:87 , srv1670556048:87 , srv1688134062:87 , srv1800606036:87 , srv1915954917:87 , srv1930341904:87 , srv1937591761:87 , srv1989299476:87 , srv469062581:87 , srv546890675:87 , srv705603278:87 , srv769610212:87 , srv83432528:87 , srv901477338:87 , srv960038509:87 , srv994592199:87 , srv1246567800:88 , srv142389443:88 , srv1823518301:88 , srv30186204:88 , srv412130545:88 , srv1049728892:89 , srv1073415045:89 , srv1790185385:89 , srv2099748639:89 , srv502888393:90 , srv1811556060:528 } 2023-10-21 23:53:30,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:30,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv469062581=68, srv1220181173=8, srv1309990787=15, srv687535454=83, srv213003577=59, srv1120430302=6, srv1533237564=24, srv994592199=99, srv2099748639=57, srv453041801=66, srv1800606036=38, srv1811556060=39, srv1096612029=5, srv412130545=65, srv1666472927=29, srv1095557362=4, srv831407119=92, srv1453843232=21, srv54995219=74, srv587985337=78, srv631059494=80, srv1322589169=16, srv1670556048=30, srv864550131=94, srv30186204=63, srv519445519=71, srv545675838=72, srv1229662897=9, srv158974173=25, srv145339652=20, srv901477338=95, srv1889694851=42, srv266101260=61, srv2091462929=56, srv142389443=19, srv331736176=64, srv1937591761=47, srv1989299476=51, srv745625445=87, srv769610212=90, srv705603278=84, srv2014465970=53, srv502888393=70, srv743137411=86, srv1417244692=18, srv624626249=79, srv1049728892=1, srv1766169289=36, srv234283435=60, srv1073415045=2, srv2015632305=54, srv2066275064=55, srv1246567800=10, srv1517615689=23, srv190063787=43, srv551407204=75, srv147741240=22, srv1972075018=49, srv1019185234=0, srv1903933800=44, srv586356351=77, srv679515436=82, srv960038509=97, srv457089955=67, srv300753910=62, srv1305873640=14, srv75901707=88, srv723936452=85, srv1698101199=32, srv2007893908=52, srv1930341904=46, srv1274638483=11, srv1966705335=48, srv1915954917=45, srv1688134062=31, srv485930826=69, srv1290924388=13, srv175258404=34, srv1085306726=3, srv1284107032=12, srv555982848=76, srv546890675=73, srv83432528=93, srv1790185385=37, srv1823518301=40, srv1977912307=50, srv1735410586=33, srv953988696=96, srv2120521873=58, srv1868591960=41, srv1602079367=26, srv1326286179=17, srv6384212=81, srv113810627=7, srv176547975=35, srv805630119=91, srv1655297088=28, srv1639527342=27, srv769249723=89, srv978064746=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-21 23:53:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-21 23:53:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-21 23:53:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-21 23:53:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-21 23:53:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-21 23:53:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-21 23:53:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-21 23:53:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-21 23:53:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-21 23:53:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-21 23:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-21 23:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-21 23:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-21 23:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-21 23:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-21 23:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-21 23:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-21 23:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-21 23:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-21 23:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-21 23:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-21 23:53:30,592 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006547761159855293); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006598983018610534); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006547761159855293); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:30,593 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-21 23:53:30,593 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.5895589302761E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006547761159855293); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006598983018610534); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006547761159855293); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-21 23:53:31,021 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 533 ms to try 5000 different iterations. Found a solution that moves 265 regions; Going from a computed imbalance of 7.5895589302761E-4 to a new imbalance of 2.854917141273435E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.029444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:31,023 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1533237564:83 , srv2099748639:83 , srv213003577:83 , srv30186204:83 , srv502888393:83 , srv687535454:83 , srv864550131:83 , srv960038509:83 , srv1073415045:84 , srv1085306726:84 , srv1095557362:84 , srv1096612029:84 , srv1246567800:84 , srv1274638483:84 , srv142389443:84 , srv147741240:84 , srv1517615689:84 , srv1639527342:84 , srv1670556048:84 , srv1688134062:84 , srv175258404:84 , srv176547975:84 , srv1766169289:84 , srv1790185385:84 , srv1800606036:84 , srv1823518301:84 , srv1889694851:84 , srv1937591761:84 , srv2007893908:84 , srv2120521873:84 , srv300753910:84 , srv453041801:84 , srv485930826:84 , srv54995219:84 , srv551407204:84 , srv723936452:84 , srv745625445:84 , srv769249723:84 , srv769610212:84 , srv978064746:84 , srv1019185234:85 , srv1049728892:85 , srv1120430302:85 , srv113810627:85 , srv1220181173:85 , srv1229662897:85 , srv1305873640:85 , srv1309990787:85 , srv1322589169:85 , srv1326286179:85 , srv1417244692:85 , srv145339652:85 , srv1453843232:85 , srv158974173:85 , srv1602079367:85 , srv1655297088:85 , srv1666472927:85 , srv1698101199:85 , srv1735410586:85 , srv1868591960:85 , srv190063787:85 , srv1903933800:85 , srv1930341904:85 , srv1966705335:85 , srv1972075018:85 , srv1977912307:85 , srv1989299476:85 , srv2014465970:85 , srv2015632305:85 , srv2091462929:85 , srv234283435:85 , srv266101260:85 , srv331736176:85 , srv412130545:85 , srv457089955:85 , srv469062581:85 , srv519445519:85 , srv545675838:85 , srv546890675:85 , srv555982848:85 , srv587985337:85 , srv624626249:85 , srv6384212:85 , srv679515436:85 , srv705603278:85 , srv743137411:85 , srv805630119:85 , srv831407119:85 , srv83432528:85 , srv901477338:85 , srv953988696:85 , srv994592199:85 , srv1284107032:86 , srv1290924388:86 , srv1915954917:86 , srv2066275064:86 , srv586356351:86 , srv631059494:86 , srv75901707:86 , srv1811556060:626 } 2023-10-21 23:53:31,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:31,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv469062581=68, srv1220181173=8, srv1309990787=15, srv687535454=83, srv213003577=59, srv1120430302=6, srv1533237564=24, srv994592199=99, srv2099748639=57, srv453041801=66, srv1800606036=38, srv1811556060=39, srv1096612029=5, srv412130545=65, srv1666472927=29, srv1095557362=4, srv831407119=92, srv1453843232=21, srv54995219=74, srv587985337=78, srv631059494=80, srv1322589169=16, srv1670556048=30, srv864550131=94, srv30186204=63, srv519445519=71, srv545675838=72, srv1229662897=9, srv158974173=25, srv145339652=20, srv901477338=95, srv1889694851=42, srv266101260=61, srv2091462929=56, srv142389443=19, srv331736176=64, srv1937591761=47, srv1989299476=51, srv745625445=87, srv769610212=90, srv705603278=84, srv2014465970=53, srv502888393=70, srv743137411=86, srv1417244692=18, srv624626249=79, srv1049728892=1, srv1766169289=36, srv234283435=60, srv1073415045=2, srv2015632305=54, srv2066275064=55, srv1246567800=10, srv1517615689=23, srv190063787=43, srv551407204=75, srv147741240=22, srv1972075018=49, srv1019185234=0, srv1903933800=44, srv586356351=77, srv679515436=82, srv960038509=97, srv457089955=67, srv300753910=62, srv1305873640=14, srv75901707=88, srv723936452=85, srv1698101199=32, srv2007893908=52, srv1930341904=46, srv1274638483=11, srv1966705335=48, srv1915954917=45, srv1688134062=31, srv485930826=69, srv1290924388=13, srv175258404=34, srv1085306726=3, srv1284107032=12, srv555982848=76, srv546890675=73, srv83432528=93, srv1790185385=37, srv1823518301=40, srv1977912307=50, srv1735410586=33, srv953988696=96, srv2120521873=58, srv1868591960=41, srv1602079367=26, srv1326286179=17, srv6384212=81, srv113810627=7, srv176547975=35, srv805630119=91, srv1655297088=28, srv1639527342=27, srv769249723=89, srv978064746=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:31,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:31,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-21 23:53:31,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-21 23:53:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-21 23:53:31,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-21 23:53:31,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-21 23:53:31,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-21 23:53:31,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:31,170 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-21 23:53:31,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6693601752848763E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-21 23:53:31,450 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 390 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 2.6693601752848763E-5 to a new imbalance of 7.583817690791035E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:31,450 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1220181173:83 , srv1698101199:83 , srv1915954917:83 , srv1977912307:83 , srv412130545:83 , srv631059494:83 , srv679515436:83 , srv75901707:83 , srv1019185234:84 , srv1049728892:84 , srv1073415045:84 , srv1085306726:84 , srv1095557362:84 , srv1096612029:84 , srv1120430302:84 , srv113810627:84 , srv1229662897:84 , srv1246567800:84 , srv1274638483:84 , srv1284107032:84 , srv1290924388:84 , srv1305873640:84 , srv1309990787:84 , srv1322589169:84 , srv1326286179:84 , srv1417244692:84 , srv142389443:84 , srv145339652:84 , srv1453843232:84 , srv147741240:84 , srv1517615689:84 , srv1533237564:84 , srv158974173:84 , srv1602079367:84 , srv1655297088:84 , srv1666472927:84 , srv1670556048:84 , srv1688134062:84 , srv1735410586:84 , srv175258404:84 , srv1766169289:84 , srv1790185385:84 , srv1800606036:84 , srv1868591960:84 , srv1889694851:84 , srv190063787:84 , srv1903933800:84 , srv1930341904:84 , srv1937591761:84 , srv1966705335:84 , srv1972075018:84 , srv1989299476:84 , srv2007893908:84 , srv2014465970:84 , srv2015632305:84 , srv2066275064:84 , srv2091462929:84 , srv2120521873:84 , srv234283435:84 , srv266101260:84 , srv300753910:84 , srv331736176:84 , srv453041801:84 , srv457089955:84 , srv469062581:84 , srv485930826:84 , srv502888393:84 , srv519445519:84 , srv545675838:84 , srv546890675:84 , srv54995219:84 , srv551407204:84 , srv555982848:84 , srv586356351:84 , srv587985337:84 , srv624626249:84 , srv6384212:84 , srv705603278:84 , srv723936452:84 , srv743137411:84 , srv769249723:84 , srv769610212:84 , srv805630119:84 , srv831407119:84 , srv83432528:84 , srv864550131:84 , srv901477338:84 , srv953988696:84 , srv960038509:84 , srv978064746:84 , srv994592199:84 , srv1639527342:85 , srv176547975:85 , srv1823518301:85 , srv2099748639:85 , srv213003577:85 , srv30186204:85 , srv687535454:85 , srv745625445:85 , srv1811556060:684 } 2023-10-21 23:53:31,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:31,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv469062581=68, srv1220181173=8, srv1309990787=15, srv687535454=83, srv213003577=59, srv1120430302=6, srv1533237564=24, srv994592199=99, srv2099748639=57, srv453041801=66, srv1800606036=38, srv1811556060=39, srv1096612029=5, srv412130545=65, srv1666472927=29, srv1095557362=4, srv831407119=92, srv1453843232=21, srv54995219=74, srv587985337=78, srv631059494=80, srv1322589169=16, srv1670556048=30, srv864550131=94, srv30186204=63, srv519445519=71, srv545675838=72, srv1229662897=9, srv158974173=25, srv145339652=20, srv901477338=95, srv1889694851=42, srv266101260=61, srv2091462929=56, srv142389443=19, srv331736176=64, srv1937591761=47, srv1989299476=51, srv745625445=87, srv769610212=90, srv705603278=84, srv2014465970=53, srv502888393=70, srv743137411=86, srv1417244692=18, srv624626249=79, srv1049728892=1, srv1766169289=36, srv234283435=60, srv1073415045=2, srv2015632305=54, srv2066275064=55, srv1246567800=10, srv1517615689=23, srv190063787=43, srv551407204=75, srv147741240=22, srv1972075018=49, srv1019185234=0, srv1903933800=44, srv586356351=77, srv679515436=82, srv960038509=97, srv457089955=67, srv300753910=62, srv1305873640=14, srv75901707=88, srv723936452=85, srv1698101199=32, srv2007893908=52, srv1930341904=46, srv1274638483=11, srv1966705335=48, srv1915954917=45, srv1688134062=31, srv485930826=69, srv1290924388=13, srv175258404=34, srv1085306726=3, srv1284107032=12, srv555982848=76, srv546890675=73, srv83432528=93, srv1790185385=37, srv1823518301=40, srv1977912307=50, srv1735410586=33, srv953988696=96, srv2120521873=58, srv1868591960=41, srv1602079367=26, srv1326286179=17, srv6384212=81, srv113810627=7, srv176547975=35, srv805630119=91, srv1655297088=28, srv1639527342=27, srv769249723=89, srv978064746=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:31,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-21 23:53:31,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-21 23:53:31,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-21 23:53:31,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-21 23:53:31,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-21 23:53:31,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-21 23:53:31,584 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:31,585 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1220181173:83 , srv1698101199:83 , srv1915954917:83 , srv1977912307:83 , srv412130545:83 , srv631059494:83 , srv679515436:83 , srv75901707:83 , srv1019185234:84 , srv1049728892:84 , srv1073415045:84 , srv1085306726:84 , srv1095557362:84 , srv1096612029:84 , srv1120430302:84 , srv113810627:84 , srv1229662897:84 , srv1246567800:84 , srv1274638483:84 , srv1284107032:84 , srv1290924388:84 , srv1305873640:84 , srv1309990787:84 , srv1322589169:84 , srv1326286179:84 , srv1417244692:84 , srv142389443:84 , srv145339652:84 , srv1453843232:84 , srv147741240:84 , srv1517615689:84 , srv1533237564:84 , srv158974173:84 , srv1602079367:84 , srv1655297088:84 , srv1666472927:84 , srv1670556048:84 , srv1688134062:84 , srv1735410586:84 , srv175258404:84 , srv1766169289:84 , srv1790185385:84 , srv1800606036:84 , srv1868591960:84 , srv1889694851:84 , srv190063787:84 , srv1903933800:84 , srv1930341904:84 , srv1937591761:84 , srv1966705335:84 , srv1972075018:84 , srv1989299476:84 , srv2007893908:84 , srv2014465970:84 , srv2015632305:84 , srv2066275064:84 , srv2091462929:84 , srv2120521873:84 , srv234283435:84 , srv266101260:84 , srv300753910:84 , srv331736176:84 , srv453041801:84 , srv457089955:84 , srv469062581:84 , srv485930826:84 , srv502888393:84 , srv519445519:84 , srv545675838:84 , srv546890675:84 , srv54995219:84 , srv551407204:84 , srv555982848:84 , srv586356351:84 , srv587985337:84 , srv624626249:84 , srv6384212:84 , srv705603278:84 , srv723936452:84 , srv743137411:84 , srv769249723:84 , srv769610212:84 , srv805630119:84 , srv831407119:84 , srv83432528:84 , srv864550131:84 , srv901477338:84 , srv953988696:84 , srv960038509:84 , srv978064746:84 , srv994592199:84 , srv1639527342:85 , srv176547975:85 , srv1823518301:85 , srv2099748639:85 , srv213003577:85 , srv30186204:85 , srv687535454:85 , srv745625445:85 , srv1811556060:684 } 2023-10-21 23:53:31,613 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=386 (was 386), ProcessCount=171 (was 171), AvailableMemoryMB=8174 (was 8695) 2023-10-21 23:53:31,620 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=171, AvailableMemoryMB=8174 2023-10-21 23:53:31,621 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-21 23:53:31,621 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-21 23:53:31,635 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1043493328:84 , srv1057141791:84 , srv1063665029:84 , srv1066279861:84 , srv1326882646:84 , srv1341920545:84 , srv1400150536:84 , srv1410486297:84 , srv1433003553:84 , srv1544531868:84 , srv1630168892:84 , srv1648022660:84 , srv1726484195:84 , srv197643484:84 , srv2073923579:84 , srv305188815:84 , srv378006069:84 , srv39155258:84 , srv48468366:84 , srv524676448:84 , srv656723150:84 , srv658060852:84 , srv773662086:84 , srv783488040:84 , srv804514326:84 , srv822738981:84 , srv828165114:84 , srv882881739:84 , srv934054063:84 , srv1603343170:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-21 23:53:31,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:53:31,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2073923579=15, srv656723150=21, srv1603343170=10, srv934054063=29, srv1341920545=5, srv882881739=28, srv658060852=22, srv524676448=20, srv1433003553=8, srv1410486297=7, srv1544531868=9, srv1043493328=0, srv1648022660=12, srv822738981=26, srv48468366=19, srv197643484=14, srv773662086=23, srv1630168892=11, srv1726484195=13, srv804514326=25, srv783488040=24, srv1063665029=2, srv1057141791=1, srv828165114=27, srv1326882646=4, srv1066279861=3, srv305188815=16, srv39155258=18, srv1400150536=6, srv378006069=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:53:31,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:53:31,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:53:31,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-21 23:53:31,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1358864667731954, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:53:31,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991262121708398, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1358864667731954, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-21 23:55:31,657 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5612874 different iterations. Found a solution that moves 1857 regions; Going from a computed imbalance of 0.991262121708398 to a new imbalance of 4.334330639506329E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:55:31,665 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1043493328:90 , srv1057141791:90 , srv1063665029:90 , srv1066279861:90 , srv1326882646:90 , srv1341920545:90 , srv1400150536:90 , srv1410486297:90 , srv1433003553:90 , srv1544531868:90 , srv1603343170:90 , srv1630168892:90 , srv1648022660:90 , srv1726484195:90 , srv197643484:90 , srv2073923579:90 , srv305188815:90 , srv378006069:90 , srv39155258:90 , srv48468366:90 , srv524676448:90 , srv656723150:90 , srv658060852:90 , srv773662086:90 , srv783488040:90 , srv804514326:90 , srv822738981:90 , srv828165114:90 , srv882881739:90 , srv934054063:90 } 2023-10-21 23:55:31,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-21 23:55:31,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2073923579=15, srv656723150=21, srv1603343170=10, srv934054063=29, srv1341920545=5, srv882881739=28, srv658060852=22, srv524676448=20, srv1433003553=8, srv1410486297=7, srv1544531868=9, srv1043493328=0, srv1648022660=12, srv822738981=26, srv48468366=19, srv197643484=14, srv773662086=23, srv1630168892=11, srv1726484195=13, srv804514326=25, srv783488040=24, srv1063665029=2, srv1057141791=1, srv828165114=27, srv1326882646=4, srv1066279861=3, srv305188815=16, srv39155258=18, srv1400150536=6, srv378006069=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-21 23:55:31,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-21 23:55:31,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-21 23:55:31,686 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-21 23:55:31,686 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1043493328:90 , srv1057141791:90 , srv1063665029:90 , srv1066279861:90 , srv1326882646:90 , srv1341920545:90 , srv1400150536:90 , srv1410486297:90 , srv1433003553:90 , srv1544531868:90 , srv1603343170:90 , srv1630168892:90 , srv1648022660:90 , srv1726484195:90 , srv197643484:90 , srv2073923579:90 , srv305188815:90 , srv378006069:90 , srv39155258:90 , srv48468366:90 , srv524676448:90 , srv656723150:90 , srv658060852:90 , srv773662086:90 , srv783488040:90 , srv804514326:90 , srv822738981:90 , srv828165114:90 , srv882881739:90 , srv934054063:90 } 2023-10-21 23:55:31,700 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 386), ProcessCount=167 (was 171), AvailableMemoryMB=7892 (was 8174)