2024-01-29 07:53:11,690 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-29 07:53:12,013 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-29 07:53:12,136 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-29 07:53:12,217 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-29 07:53:12,227 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-29 07:53:12,243 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=167, AvailableMemoryMB=3438 2024-01-29 07:53:12,246 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-29 07:53:12,247 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-29 07:53:12,373 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007180284:84 , srv1021632628:84 , srv1030618870:84 , srv1106052325:84 , srv1133428586:84 , srv1150086058:84 , srv1159227854:84 , srv1184416592:84 , srv1187944245:84 , srv1222688003:84 , srv1233290615:84 , srv1233432981:84 , srv1268668407:84 , srv1269194773:84 , srv1289405825:84 , srv1291316832:84 , srv1311725501:84 , srv1341739178:84 , srv1419154418:84 , srv1420911502:84 , srv1457866758:84 , srv1481886469:84 , srv1483983950:84 , srv1490559717:84 , srv1527795970:84 , srv1558132982:84 , srv1567717588:84 , srv1590191037:84 , srv1646185025:84 , srv1652872381:84 , srv166684819:84 , srv1680376438:84 , srv1778930353:84 , srv178369566:84 , srv1792295415:84 , srv1830862280:84 , srv1840265795:84 , srv1847082882:84 , srv1874789692:84 , srv1875067101:84 , srv1900728987:84 , srv1926892629:84 , srv1950046546:84 , srv1967780600:84 , srv2006835825:84 , srv2007783588:84 , srv2049757142:84 , srv2051494283:84 , srv2074045648:84 , srv212543028:84 , srv2132922444:84 , srv2144762281:84 , srv217797046:84 , srv224264591:84 , srv23364737:84 , srv234205575:84 , srv250747545:84 , srv280502622:84 , srv320911768:84 , srv322630073:84 , srv342843675:84 , srv360482943:84 , srv375159797:84 , srv385728287:84 , srv405522297:84 , srv410822594:84 , srv464694885:84 , srv513481319:84 , srv518022677:84 , srv518668166:84 , srv544879538:84 , srv556562403:84 , srv56497672:84 , srv588053515:84 , srv603480914:84 , srv620473205:84 , srv625843453:84 , srv64253351:84 , srv659666093:84 , srv663089248:84 , srv721207660:84 , srv732601202:84 , srv736428502:84 , srv738653342:84 , srv771552259:84 , srv79804068:84 , srv834896447:84 , srv844631476:84 , srv846291124:84 , srv867680627:84 , srv868371219:84 , srv869211360:84 , srv874467828:84 , srv917992417:84 , srv920160202:84 , srv929117075:84 , srv950702581:84 , srv993093361:84 , srv996436790:84 , srv834054977:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-29 07:53:12,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:53:12,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv996436790=99, srv320911768=58, srv867680627=90, srv464694885=66, srv1419154418=18, srv917992417=94, srv1527795970=24, srv1233432981=11, srv846291124=89, srv844631476=88, srv1133428586=4, srv1680376438=31, srv1030618870=2, srv2051494283=47, srv2007783588=45, srv217797046=52, srv732601202=81, srv1233290615=10, srv1830862280=35, srv1222688003=9, srv1021632628=1, srv544879538=70, srv1874789692=38, srv950702581=97, srv1187944245=8, srv405522297=64, srv1792295415=34, srv1341739178=17, srv2006835825=44, srv360482943=61, srv64253351=77, srv79804068=85, srv518668166=69, srv1950046546=42, srv322630073=59, srv620473205=75, srv23364737=54, srv834054977=86, srv1778930353=32, srv1875067101=39, srv738653342=83, srv721207660=80, srv1558132982=25, srv1184416592=7, srv993093361=98, srv1311725501=16, srv224264591=53, srv410822594=65, srv513481319=67, srv920160202=95, srv178369566=33, srv1490559717=23, srv1652872381=29, srv1457866758=20, srv375159797=62, srv1106052325=3, srv588053515=73, srv2049757142=46, srv929117075=96, srv868371219=91, srv1567717588=26, srv1926892629=41, srv212543028=49, srv2144762281=51, srv166684819=30, srv2132922444=50, srv736428502=82, srv1291316832=15, srv1900728987=40, srv1268668407=12, srv603480914=74, srv556562403=71, srv280502622=57, srv250747545=56, srv1269194773=13, srv518022677=68, srv625843453=76, srv874467828=93, srv342843675=60, srv659666093=78, srv1646185025=28, srv1847082882=37, srv1420911502=19, srv1590191037=27, srv1483983950=22, srv1159227854=6, srv663089248=79, srv771552259=84, srv834896447=87, srv1840265795=36, srv2074045648=48, srv869211360=92, srv1967780600=43, srv385728287=63, srv234205575=55, srv56497672=72, srv1289405825=14, srv1481886469=21, srv1150086058=5, srv1007180284=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:53:12,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:53:12,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:53:12,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:53:12,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:53:12,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:53:12,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 07:53:12,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 07:53:12,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 07:53:12,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 07:53:12,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 07:53:12,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 07:53:12,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 07:53:12,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 07:53:12,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 07:53:12,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 07:53:12,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 07:53:12,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 07:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 07:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 07:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 07:53:12,748 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:12,749 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 07:53:12,749 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 07:53:13,591 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1076 ms to try 5000 different iterations. Found a solution that moves 4695 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17906952061483436. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03758716021189433, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05182202124118554, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5216666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03758716021189433, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24241666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:13,614 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1291316832:80 , srv1341739178:81 , srv2051494283:81 , srv663089248:82 , srv1558132982:83 , srv322630073:83 , srv513481319:83 , srv56497672:83 , srv929117075:83 , srv950702581:83 , srv1021632628:84 , srv1233290615:84 , srv1311725501:84 , srv1680376438:84 , srv1792295415:84 , srv342843675:84 , srv844631476:84 , srv1289405825:85 , srv1646185025:85 , srv1847082882:85 , srv1875067101:85 , srv405522297:85 , srv556562403:85 , srv721207660:85 , srv736428502:85 , srv738653342:85 , srv771552259:85 , srv868371219:85 , srv874467828:85 , srv920160202:85 , srv1106052325:86 , srv1133428586:86 , srv1187944245:86 , srv1268668407:86 , srv1269194773:86 , srv1652872381:86 , srv1778930353:86 , srv1840265795:86 , srv1874789692:86 , srv1926892629:86 , srv2006835825:86 , srv23364737:86 , srv518022677:86 , srv518668166:86 , srv588053515:86 , srv603480914:86 , srv993093361:86 , srv1007180284:87 , srv1590191037:87 , srv1950046546:87 , srv1967780600:87 , srv2007783588:87 , srv2049757142:87 , srv212543028:87 , srv217797046:87 , srv224264591:87 , srv250747545:87 , srv280502622:87 , srv464694885:87 , srv544879538:87 , srv64253351:87 , srv659666093:87 , srv834896447:87 , srv867680627:87 , srv869211360:87 , srv1150086058:88 , srv1184416592:88 , srv1222688003:88 , srv1233432981:88 , srv1419154418:88 , srv1457866758:88 , srv178369566:88 , srv2144762281:88 , srv234205575:88 , srv410822594:88 , srv620473205:88 , srv625843453:88 , srv732601202:88 , srv846291124:88 , srv917992417:88 , srv1159227854:89 , srv1481886469:89 , srv1490559717:89 , srv1567717588:89 , srv2074045648:89 , srv320911768:89 , srv360482943:89 , srv375159797:89 , srv385728287:89 , srv1483983950:90 , srv1527795970:90 , srv1900728987:90 , srv996436790:90 , srv1420911502:91 , srv1830862280:91 , srv2132922444:91 , srv1030618870:92 , srv79804068:92 , srv166684819:95 , srv834054977:424 } 2024-01-29 07:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv996436790=99, srv320911768=58, srv867680627=90, srv464694885=66, srv1419154418=18, srv917992417=94, srv1527795970=24, srv1233432981=11, srv846291124=89, srv844631476=88, srv1133428586=4, srv1680376438=31, srv1030618870=2, srv2051494283=47, srv2007783588=45, srv217797046=52, srv732601202=81, srv1233290615=10, srv1830862280=35, srv1222688003=9, srv1021632628=1, srv544879538=70, srv1874789692=38, srv950702581=97, srv1187944245=8, srv405522297=64, srv1792295415=34, srv1341739178=17, srv2006835825=44, srv360482943=61, srv64253351=77, srv79804068=85, srv518668166=69, srv1950046546=42, srv322630073=59, srv620473205=75, srv23364737=54, srv834054977=86, srv1778930353=32, srv1875067101=39, srv738653342=83, srv721207660=80, srv1558132982=25, srv1184416592=7, srv993093361=98, srv1311725501=16, srv224264591=53, srv410822594=65, srv513481319=67, srv920160202=95, srv178369566=33, srv1490559717=23, srv1652872381=29, srv1457866758=20, srv375159797=62, srv1106052325=3, srv588053515=73, srv2049757142=46, srv929117075=96, srv868371219=91, srv1567717588=26, srv1926892629=41, srv212543028=49, srv2144762281=51, srv166684819=30, srv2132922444=50, srv736428502=82, srv1291316832=15, srv1900728987=40, srv1268668407=12, srv603480914=74, srv556562403=71, srv280502622=57, srv250747545=56, srv1269194773=13, srv518022677=68, srv625843453=76, srv874467828=93, srv342843675=60, srv659666093=78, srv1646185025=28, srv1847082882=37, srv1420911502=19, srv1590191037=27, srv1483983950=22, srv1159227854=6, srv663089248=79, srv771552259=84, srv834896447=87, srv1840265795=36, srv2074045648=48, srv869211360=92, srv1967780600=43, srv385728287=63, srv234205575=55, srv56497672=72, srv1289405825=14, srv1481886469=21, srv1150086058=5, srv1007180284=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 07:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 07:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 07:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 07:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 07:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 07:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 07:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 07:53:13,804 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03758716021189433, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05182202124118554, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03758716021189433, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24241666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:13,804 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 07:53:13,804 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17903664552218093, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03758716021189433, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05182202124118554, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03758716021189433, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24241666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 07:53:14,064 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 388 ms to try 5000 different iterations. Found a solution that moves 2684 regions; Going from a computed imbalance of 0.17903664552218093 to a new imbalance of 0.015073092232183125. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020044843105623662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02959130518094815, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2982222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020044843105623662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043833333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:14,072 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv166684819:78 , srv2132922444:79 , srv79804068:80 , srv1527795970:81 , srv1830862280:81 , srv320911768:81 , srv996436790:81 , srv1420911502:82 , srv1457866758:82 , srv1481886469:82 , srv1567717588:82 , srv464694885:82 , srv620473205:82 , srv1159227854:83 , srv1184416592:83 , srv1483983950:83 , srv1490559717:83 , srv1900728987:83 , srv360482943:83 , srv375159797:83 , srv385728287:83 , srv518022677:83 , srv917992417:83 , srv1030618870:84 , srv2006835825:84 , srv2144762281:84 , srv234205575:84 , srv410822594:84 , srv1007180284:85 , srv1268668407:85 , srv1419154418:85 , srv1840265795:85 , srv1874789692:85 , srv1950046546:85 , srv1967780600:85 , srv2074045648:85 , srv212543028:85 , srv217797046:85 , srv23364737:85 , srv280502622:85 , srv603480914:85 , srv64253351:85 , srv732601202:85 , srv846291124:85 , srv867680627:85 , srv869211360:85 , srv1021632628:86 , srv1150086058:86 , srv1222688003:86 , srv1233432981:86 , srv1590191037:86 , srv178369566:86 , srv1847082882:86 , srv2007783588:86 , srv2049757142:86 , srv224264591:86 , srv250747545:86 , srv342843675:86 , srv513481319:86 , srv518668166:86 , srv625843453:86 , srv736428502:86 , srv738653342:86 , srv1187944245:87 , srv1233290615:87 , srv1269194773:87 , srv1311725501:87 , srv1646185025:87 , srv1680376438:87 , srv1778930353:87 , srv322630073:87 , srv544879538:87 , srv556562403:87 , srv659666093:87 , srv721207660:87 , srv771552259:87 , srv834896447:87 , srv920160202:87 , srv929117075:87 , srv993093361:87 , srv1106052325:88 , srv1289405825:88 , srv1652872381:88 , srv1875067101:88 , srv588053515:88 , srv663089248:88 , srv868371219:88 , srv874467828:88 , srv1133428586:89 , srv1792295415:89 , srv950702581:89 , srv1558132982:90 , srv405522297:90 , srv56497672:90 , srv844631476:90 , srv1926892629:91 , srv1291316832:92 , srv1341739178:92 , srv2051494283:92 , srv834054977:528 } 2024-01-29 07:53:14,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:53:14,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv996436790=99, srv320911768=58, srv867680627=90, srv464694885=66, srv1419154418=18, srv917992417=94, srv1527795970=24, srv1233432981=11, srv846291124=89, srv844631476=88, srv1133428586=4, srv1680376438=31, srv1030618870=2, srv2051494283=47, srv2007783588=45, srv217797046=52, srv732601202=81, srv1233290615=10, srv1830862280=35, srv1222688003=9, srv1021632628=1, srv544879538=70, srv1874789692=38, srv950702581=97, srv1187944245=8, srv405522297=64, srv1792295415=34, srv1341739178=17, srv2006835825=44, srv360482943=61, srv64253351=77, srv79804068=85, srv518668166=69, srv1950046546=42, srv322630073=59, srv620473205=75, srv23364737=54, srv834054977=86, srv1778930353=32, srv1875067101=39, srv738653342=83, srv721207660=80, srv1558132982=25, srv1184416592=7, srv993093361=98, srv1311725501=16, srv224264591=53, srv410822594=65, srv513481319=67, srv920160202=95, srv178369566=33, srv1490559717=23, srv1652872381=29, srv1457866758=20, srv375159797=62, srv1106052325=3, srv588053515=73, srv2049757142=46, srv929117075=96, srv868371219=91, srv1567717588=26, srv1926892629=41, srv212543028=49, srv2144762281=51, srv166684819=30, srv2132922444=50, srv736428502=82, srv1291316832=15, srv1900728987=40, srv1268668407=12, srv603480914=74, srv556562403=71, srv280502622=57, srv250747545=56, srv1269194773=13, srv518022677=68, srv625843453=76, srv874467828=93, srv342843675=60, srv659666093=78, srv1646185025=28, srv1847082882=37, srv1420911502=19, srv1590191037=27, srv1483983950=22, srv1159227854=6, srv663089248=79, srv771552259=84, srv834896447=87, srv1840265795=36, srv2074045648=48, srv869211360=92, srv1967780600=43, srv385728287=63, srv234205575=55, srv56497672=72, srv1289405825=14, srv1481886469=21, srv1150086058=5, srv1007180284=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:53:14,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:53:14,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:53:14,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:53:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 07:53:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 07:53:14,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 07:53:14,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 07:53:14,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 07:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 07:53:14,238 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020044843105623662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02959130518094815, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020044843105623662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043833333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:14,238 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 07:53:14,238 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015054298462495829, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020044843105623662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02959130518094815, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020044843105623662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043833333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 07:53:14,477 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 356 ms to try 5000 different iterations. Found a solution that moves 993 regions; Going from a computed imbalance of 0.015054298462495829 to a new imbalance of 6.941336294469796E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005898505584103609); MoveCostFunction : (multiplier=7.0, imbalance=0.11033333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:14,481 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1030618870:81 , srv1926892629:81 , srv1133428586:82 , srv1150086058:83 , srv178369566:83 , srv2074045648:83 , srv405522297:83 , srv588053515:83 , srv659666093:83 , srv1106052325:84 , srv1222688003:84 , srv1233432981:84 , srv1341739178:84 , srv1419154418:84 , srv1420911502:84 , srv1558132982:84 , srv1652872381:84 , srv1792295415:84 , srv1900728987:84 , srv2051494283:84 , srv224264591:84 , srv250747545:84 , srv544879538:84 , srv56497672:84 , srv625843453:84 , srv732601202:84 , srv834896447:84 , srv844631476:84 , srv868371219:84 , srv874467828:84 , srv993093361:84 , srv1007180284:85 , srv1159227854:85 , srv1187944245:85 , srv1289405825:85 , srv1481886469:85 , srv1590191037:85 , srv166684819:85 , srv1778930353:85 , srv1875067101:85 , srv1950046546:85 , srv2007783588:85 , srv2049757142:85 , srv212543028:85 , srv217797046:85 , srv410822594:85 , srv518668166:85 , srv64253351:85 , srv721207660:85 , srv846291124:85 , srv920160202:85 , srv1269194773:86 , srv1291316832:86 , srv1483983950:86 , srv1490559717:86 , srv1646185025:86 , srv1680376438:86 , srv1830862280:86 , srv1840265795:86 , srv1874789692:86 , srv1967780600:86 , srv2006835825:86 , srv2144762281:86 , srv234205575:86 , srv280502622:86 , srv360482943:86 , srv375159797:86 , srv385728287:86 , srv556562403:86 , srv603480914:86 , srv771552259:86 , srv79804068:86 , srv867680627:86 , srv869211360:86 , srv917992417:86 , srv950702581:86 , srv1021632628:87 , srv1184416592:87 , srv1233290615:87 , srv1268668407:87 , srv1311725501:87 , srv1527795970:87 , srv1567717588:87 , srv23364737:87 , srv322630073:87 , srv342843675:87 , srv663089248:87 , srv736428502:87 , srv738653342:87 , srv1457866758:88 , srv1847082882:88 , srv2132922444:88 , srv320911768:88 , srv464694885:88 , srv518022677:88 , srv620473205:88 , srv929117075:88 , srv996436790:88 , srv513481319:89 , srv834054977:548 } 2024-01-29 07:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv996436790=99, srv320911768=58, srv867680627=90, srv464694885=66, srv1419154418=18, srv917992417=94, srv1527795970=24, srv1233432981=11, srv846291124=89, srv844631476=88, srv1133428586=4, srv1680376438=31, srv1030618870=2, srv2051494283=47, srv2007783588=45, srv217797046=52, srv732601202=81, srv1233290615=10, srv1830862280=35, srv1222688003=9, srv1021632628=1, srv544879538=70, srv1874789692=38, srv950702581=97, srv1187944245=8, srv405522297=64, srv1792295415=34, srv1341739178=17, srv2006835825=44, srv360482943=61, srv64253351=77, srv79804068=85, srv518668166=69, srv1950046546=42, srv322630073=59, srv620473205=75, srv23364737=54, srv834054977=86, srv1778930353=32, srv1875067101=39, srv738653342=83, srv721207660=80, srv1558132982=25, srv1184416592=7, srv993093361=98, srv1311725501=16, srv224264591=53, srv410822594=65, srv513481319=67, srv920160202=95, srv178369566=33, srv1490559717=23, srv1652872381=29, srv1457866758=20, srv375159797=62, srv1106052325=3, srv588053515=73, srv2049757142=46, srv929117075=96, srv868371219=91, srv1567717588=26, srv1926892629=41, srv212543028=49, srv2144762281=51, srv166684819=30, srv2132922444=50, srv736428502=82, srv1291316832=15, srv1900728987=40, srv1268668407=12, srv603480914=74, srv556562403=71, srv280502622=57, srv250747545=56, srv1269194773=13, srv518022677=68, srv625843453=76, srv874467828=93, srv342843675=60, srv659666093=78, srv1646185025=28, srv1847082882=37, srv1420911502=19, srv1590191037=27, srv1483983950=22, srv1159227854=6, srv663089248=79, srv771552259=84, srv834896447=87, srv1840265795=36, srv2074045648=48, srv869211360=92, srv1967780600=43, srv385728287=63, srv234205575=55, srv56497672=72, srv1289405825=14, srv1481886469=21, srv1150086058=5, srv1007180284=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:53:14,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:53:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 07:53:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 07:53:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 07:53:14,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 07:53:14,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 07:53:14,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 07:53:14,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 07:53:14,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 07:53:14,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 07:53:14,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 07:53:14,665 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005898505584103609); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:14,665 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 07:53:14,666 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.871804948346535E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005898505584103609); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 07:53:15,022 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 495 ms to try 5000 different iterations. Found a solution that moves 214 regions; Going from a computed imbalance of 6.871804948346535E-4 to a new imbalance of 1.2807006857467793E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.023777777777777776); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:15,023 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1269194773:83 , srv1483983950:83 , srv1567717588:83 , srv166684819:83 , srv1847082882:83 , srv1875067101:83 , srv2049757142:83 , srv996436790:83 , srv1133428586:84 , srv1184416592:84 , srv1187944245:84 , srv1222688003:84 , srv1233290615:84 , srv1233432981:84 , srv1268668407:84 , srv1289405825:84 , srv1291316832:84 , srv1311725501:84 , srv1420911502:84 , srv1457866758:84 , srv1490559717:84 , srv1527795970:84 , srv1590191037:84 , srv1646185025:84 , srv1778930353:84 , srv1830862280:84 , srv1926892629:84 , srv1967780600:84 , srv2007783588:84 , srv2132922444:84 , srv2144762281:84 , srv23364737:84 , srv234205575:84 , srv320911768:84 , srv322630073:84 , srv360482943:84 , srv375159797:84 , srv385728287:84 , srv405522297:84 , srv513481319:84 , srv544879538:84 , srv556562403:84 , srv620473205:84 , srv625843453:84 , srv736428502:84 , srv738653342:84 , srv771552259:84 , srv79804068:84 , srv844631476:84 , srv846291124:84 , srv867680627:84 , srv869211360:84 , srv929117075:84 , srv950702581:84 , srv1007180284:85 , srv1030618870:85 , srv1106052325:85 , srv1150086058:85 , srv1159227854:85 , srv1341739178:85 , srv1419154418:85 , srv1558132982:85 , srv1652872381:85 , srv1680376438:85 , srv178369566:85 , srv1840265795:85 , srv1874789692:85 , srv1900728987:85 , srv1950046546:85 , srv2074045648:85 , srv212543028:85 , srv217797046:85 , srv224264591:85 , srv250747545:85 , srv280502622:85 , srv342843675:85 , srv410822594:85 , srv464694885:85 , srv518022677:85 , srv518668166:85 , srv56497672:85 , srv588053515:85 , srv603480914:85 , srv64253351:85 , srv659666093:85 , srv663089248:85 , srv721207660:85 , srv732601202:85 , srv834896447:85 , srv868371219:85 , srv874467828:85 , srv917992417:85 , srv920160202:85 , srv993093361:85 , srv1021632628:86 , srv1481886469:86 , srv1792295415:86 , srv2006835825:86 , srv2051494283:86 , srv834054977:642 } 2024-01-29 07:53:15,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv996436790=99, srv320911768=58, srv867680627=90, srv464694885=66, srv1419154418=18, srv917992417=94, srv1527795970=24, srv1233432981=11, srv846291124=89, srv844631476=88, srv1133428586=4, srv1680376438=31, srv1030618870=2, srv2051494283=47, srv2007783588=45, srv217797046=52, srv732601202=81, srv1233290615=10, srv1830862280=35, srv1222688003=9, srv1021632628=1, srv544879538=70, srv1874789692=38, srv950702581=97, srv1187944245=8, srv405522297=64, srv1792295415=34, srv1341739178=17, srv2006835825=44, srv360482943=61, srv64253351=77, srv79804068=85, srv518668166=69, srv1950046546=42, srv322630073=59, srv620473205=75, srv23364737=54, srv834054977=86, srv1778930353=32, srv1875067101=39, srv738653342=83, srv721207660=80, srv1558132982=25, srv1184416592=7, srv993093361=98, srv1311725501=16, srv224264591=53, srv410822594=65, srv513481319=67, srv920160202=95, srv178369566=33, srv1490559717=23, srv1652872381=29, srv1457866758=20, srv375159797=62, srv1106052325=3, srv588053515=73, srv2049757142=46, srv929117075=96, srv868371219=91, srv1567717588=26, srv1926892629=41, srv212543028=49, srv2144762281=51, srv166684819=30, srv2132922444=50, srv736428502=82, srv1291316832=15, srv1900728987=40, srv1268668407=12, srv603480914=74, srv556562403=71, srv280502622=57, srv250747545=56, srv1269194773=13, srv518022677=68, srv625843453=76, srv874467828=93, srv342843675=60, srv659666093=78, srv1646185025=28, srv1847082882=37, srv1420911502=19, srv1590191037=27, srv1483983950=22, srv1159227854=6, srv663089248=79, srv771552259=84, srv834896447=87, srv1840265795=36, srv2074045648=48, srv869211360=92, srv1967780600=43, srv385728287=63, srv234205575=55, srv56497672=72, srv1289405825=14, srv1481886469=21, srv1150086058=5, srv1007180284=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 07:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 07:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 07:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 07:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 07:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 07:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 07:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 07:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 07:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 07:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 07:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 07:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 07:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 07:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 07:53:15,173 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1308546830239433E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:15,174 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1269194773:83 , srv1483983950:83 , srv1567717588:83 , srv166684819:83 , srv1847082882:83 , srv1875067101:83 , srv2049757142:83 , srv996436790:83 , srv1133428586:84 , srv1184416592:84 , srv1187944245:84 , srv1222688003:84 , srv1233290615:84 , srv1233432981:84 , srv1268668407:84 , srv1289405825:84 , srv1291316832:84 , srv1311725501:84 , srv1420911502:84 , srv1457866758:84 , srv1490559717:84 , srv1527795970:84 , srv1590191037:84 , srv1646185025:84 , srv1778930353:84 , srv1830862280:84 , srv1926892629:84 , srv1967780600:84 , srv2007783588:84 , srv2132922444:84 , srv2144762281:84 , srv23364737:84 , srv234205575:84 , srv320911768:84 , srv322630073:84 , srv360482943:84 , srv375159797:84 , srv385728287:84 , srv405522297:84 , srv513481319:84 , srv544879538:84 , srv556562403:84 , srv620473205:84 , srv625843453:84 , srv736428502:84 , srv738653342:84 , srv771552259:84 , srv79804068:84 , srv844631476:84 , srv846291124:84 , srv867680627:84 , srv869211360:84 , srv929117075:84 , srv950702581:84 , srv1007180284:85 , srv1030618870:85 , srv1106052325:85 , srv1150086058:85 , srv1159227854:85 , srv1341739178:85 , srv1419154418:85 , srv1558132982:85 , srv1652872381:85 , srv1680376438:85 , srv178369566:85 , srv1840265795:85 , srv1874789692:85 , srv1900728987:85 , srv1950046546:85 , srv2074045648:85 , srv212543028:85 , srv217797046:85 , srv224264591:85 , srv250747545:85 , srv280502622:85 , srv342843675:85 , srv410822594:85 , srv464694885:85 , srv518022677:85 , srv518668166:85 , srv56497672:85 , srv588053515:85 , srv603480914:85 , srv64253351:85 , srv659666093:85 , srv663089248:85 , srv721207660:85 , srv732601202:85 , srv834896447:85 , srv868371219:85 , srv874467828:85 , srv917992417:85 , srv920160202:85 , srv993093361:85 , srv1021632628:86 , srv1481886469:86 , srv1792295415:86 , srv2006835825:86 , srv2051494283:86 , srv834054977:642 } 2024-01-29 07:53:15,204 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=298 (was 272) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2621 (was 3438) 2024-01-29 07:53:15,213 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=167, AvailableMemoryMB=2610 2024-01-29 07:53:15,214 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-29 07:53:15,214 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-29 07:53:15,241 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1086046455:84 , srv1162040075:84 , srv120520409:84 , srv127798291:84 , srv1302483722:84 , srv1384034238:84 , srv1524682451:84 , srv1531761164:84 , srv153685932:84 , srv1599958088:84 , srv1625024602:84 , srv1631549229:84 , srv1653806088:84 , srv1748302232:84 , srv1813083285:84 , srv1891585447:84 , srv1926910431:84 , srv2017413053:84 , srv2044178512:84 , srv334276344:84 , srv375682436:84 , srv40223900:84 , srv470096398:84 , srv601761452:84 , srv665748179:84 , srv690973885:84 , srv941141538:84 , srv981951458:84 , srv985729845:84 , srv1172428723:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-29 07:53:15,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:53:15,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1086046455=0, srv120520409=3, srv1599958088=10, srv1531761164=8, srv1653806088=13, srv1891585447=16, srv985729845=29, srv1172428723=2, srv1631549229=12, srv334276344=20, srv470096398=23, srv941141538=27, srv1524682451=7, srv1813083285=15, srv1384034238=6, srv153685932=9, srv40223900=22, srv1926910431=17, srv1748302232=14, srv601761452=24, srv375682436=21, srv1625024602=11, srv1302483722=5, srv981951458=28, srv1162040075=1, srv690973885=26, srv127798291=4, srv2017413053=18, srv2044178512=19, srv665748179=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-29 07:53:15,283 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1350229622996804, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:53:15,284 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912618496209581, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1350229622996804, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-29 07:55:15,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 7183836 different iterations. Found a solution that moves 1835 regions; Going from a computed imbalance of 0.9912618496209581 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:55:15,267 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1086046455:90 , srv1162040075:90 , srv1172428723:90 , srv120520409:90 , srv127798291:90 , srv1302483722:90 , srv1384034238:90 , srv1524682451:90 , srv1531761164:90 , srv153685932:90 , srv1599958088:90 , srv1625024602:90 , srv1631549229:90 , srv1653806088:90 , srv1748302232:90 , srv1813083285:90 , srv1891585447:90 , srv1926910431:90 , srv2017413053:90 , srv2044178512:90 , srv334276344:90 , srv375682436:90 , srv40223900:90 , srv470096398:90 , srv601761452:90 , srv665748179:90 , srv690973885:90 , srv941141538:90 , srv981951458:90 , srv985729845:90 } 2024-01-29 07:55:15,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 07:55:15,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1086046455=0, srv120520409=3, srv1599958088=10, srv1531761164=8, srv1653806088=13, srv1891585447=16, srv985729845=29, srv1172428723=2, srv1631549229=12, srv334276344=20, srv470096398=23, srv941141538=27, srv1524682451=7, srv1813083285=15, srv1384034238=6, srv153685932=9, srv40223900=22, srv1926910431=17, srv1748302232=14, srv601761452=24, srv375682436=21, srv1625024602=11, srv1302483722=5, srv981951458=28, srv1162040075=1, srv690973885=26, srv127798291=4, srv2017413053=18, srv2044178512=19, srv665748179=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 07:55:15,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 07:55:15,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 07:55:15,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 07:55:15,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 07:55:15,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 07:55:15,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 07:55:15,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 07:55:15,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 07:55:15,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-29 07:55:15,286 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 07:55:15,286 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1086046455:90 , srv1162040075:90 , srv1172428723:90 , srv120520409:90 , srv127798291:90 , srv1302483722:90 , srv1384034238:90 , srv1524682451:90 , srv1531761164:90 , srv153685932:90 , srv1599958088:90 , srv1625024602:90 , srv1631549229:90 , srv1653806088:90 , srv1748302232:90 , srv1813083285:90 , srv1891585447:90 , srv1926910431:90 , srv2017413053:90 , srv2044178512:90 , srv334276344:90 , srv375682436:90 , srv40223900:90 , srv470096398:90 , srv601761452:90 , srv665748179:90 , srv690973885:90 , srv941141538:90 , srv981951458:90 , srv985729845:90 } 2024-01-29 07:55:15,298 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=187 (was 298), ProcessCount=167 (was 167), AvailableMemoryMB=2190 (was 2610)