2023-04-29 21:53:32,382 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-29 21:53:32,737 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-29 21:53:32,857 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-29 21:53:32,924 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-29 21:53:32,932 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-29 21:53:32,947 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=321, ProcessCount=171, AvailableMemoryMB=11422 2023-04-29 21:53:32,949 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-29 21:53:32,949 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-29 21:53:33,096 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1043364553:84 , srv1074747057:84 , srv108110810:84 , srv1088020356:84 , srv1113767206:84 , srv1125581439:84 , srv1128653748:84 , srv1132738059:84 , srv1175936748:84 , srv1255692040:84 , srv1274385078:84 , srv1280376766:84 , srv1349459687:84 , srv1387724456:84 , srv1391716564:84 , srv1414036783:84 , srv1423194977:84 , srv1426608652:84 , srv1427287819:84 , srv1454271534:84 , srv145800934:84 , srv1484135669:84 , srv1535359165:84 , srv1540499464:84 , srv1576350282:84 , srv1609860167:84 , srv1624135761:84 , srv1714356353:84 , srv1721279792:84 , srv1725466141:84 , srv1737490397:84 , srv1743720058:84 , srv1753123535:84 , srv1767870374:84 , srv1782743105:84 , srv180895636:84 , srv1826565413:84 , srv1838823333:84 , srv1843076639:84 , srv1853175199:84 , srv1869974786:84 , srv1872427821:84 , srv1876934701:84 , srv1884620558:84 , srv1899056738:84 , srv1943779189:84 , srv1952642552:84 , srv1957385070:84 , srv1964127812:84 , srv1970025774:84 , srv1975796237:84 , srv1996957541:84 , srv1997401603:84 , srv2002688569:84 , srv2032896893:84 , srv2036993192:84 , srv2051197484:84 , srv2063310880:84 , srv2070327025:84 , srv2115559353:84 , srv2141102789:84 , srv279865199:84 , srv281713930:84 , srv290239679:84 , srv294328551:84 , srv303296620:84 , srv303575484:84 , srv33699245:84 , srv344151307:84 , srv349234357:84 , srv382834144:84 , srv431065617:84 , srv45139609:84 , srv453517217:84 , srv469732788:84 , srv472394032:84 , srv477929048:84 , srv532835385:84 , srv53631372:84 , srv585387119:84 , srv62431695:84 , srv635831529:84 , srv65161275:84 , srv662436256:84 , srv70718470:84 , srv728275827:84 , srv748748971:84 , srv802363243:84 , srv843820921:84 , srv913854871:84 , srv915605857:84 , srv95633208:84 , srv964924983:84 , srv966008921:84 , srv974517645:84 , srv974591432:84 , srv975311096:84 , srv976159865:84 , srv984223255:84 , srv84159017:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-29 21:53:33,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:33,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1872427821=41, srv1255692040=9, srv453517217=73, srv65161275=82, srv585387119=79, srv635831529=81, srv290239679=63, srv1175936748=8, srv1884620558=43, srv279865199=61, srv1113767206=4, srv1996957541=51, srv843820921=89, srv802363243=87, srv382834144=70, srv1423194977=16, srv95633208=92, srv2141102789=60, srv964924983=93, srv532835385=77, srv344151307=68, srv1454271534=19, srv2002688569=53, srv45139609=72, srv477929048=76, srv974591432=96, srv1349459687=12, srv728275827=85, srv1074747057=1, srv84159017=88, srv1414036783=15, srv1767870374=33, srv1782743105=34, srv1274385078=10, srv1624135761=26, srv662436256=83, srv1843076639=38, srv1952642552=46, srv1737490397=30, srv53631372=78, srv976159865=98, srv1576350282=24, srv1128653748=6, srv1869974786=40, srv1899056738=44, srv1535359165=22, srv349234357=69, srv1876934701=42, srv2032896893=54, srv303296620=65, srv974517645=95, srv1964127812=48, srv469732788=74, srv2063310880=57, srv1975796237=50, srv984223255=99, srv1753123535=32, srv1721279792=28, srv1484135669=21, srv2115559353=59, srv1970025774=49, srv748748971=86, srv180895636=35, srv1540499464=23, srv294328551=64, srv1125581439=5, srv70718470=84, srv966008921=94, srv303575484=66, srv1280376766=11, srv1088020356=3, srv281713930=62, srv1427287819=18, srv62431695=80, srv1043364553=0, srv1714356353=27, srv1826565413=36, srv1838823333=37, srv1426608652=17, srv2070327025=58, srv145800934=20, srv431065617=71, srv2051197484=56, srv1725466141=29, srv975311096=97, srv1132738059=7, srv1943779189=45, srv2036993192=55, srv913854871=90, srv1609860167=25, srv1997401603=52, srv1391716564=14, srv915605857=91, srv1387724456=13, srv1743720058=31, srv1957385070=47, srv33699245=67, srv472394032=75, srv1853175199=39, srv108110810=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:33,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:33,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:33,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-29 21:53:33,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-29 21:53:33,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-29 21:53:33,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-29 21:53:33,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-29 21:53:33,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-29 21:53:33,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-29 21:53:33,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-29 21:53:33,694 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:33,695 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-29 21:53:33,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-29 21:53:34,354 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1107 ms to try 5000 different iterations. Found a solution that moves 4673 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18100613279659844. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03974196608514426, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056443410473310224, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5192222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03974196608514426, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17633333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24191666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:34,373 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv180895636:81 , srv1872427821:81 , srv2036993192:81 , srv290239679:81 , srv279865199:82 , srv966008921:82 , srv45139609:83 , srv532835385:83 , srv728275827:83 , srv974591432:83 , srv1074747057:84 , srv1387724456:84 , srv1535359165:84 , srv1714356353:84 , srv1975796237:84 , srv349234357:84 , srv469732788:84 , srv65161275:84 , srv662436256:84 , srv802363243:84 , srv915605857:84 , srv974517645:84 , srv1043364553:85 , srv1128653748:85 , srv1414036783:85 , srv1484135669:85 , srv1737490397:85 , srv1843076639:85 , srv1876934701:85 , srv1943779189:85 , srv1964127812:85 , srv294328551:85 , srv431065617:85 , srv453517217:85 , srv472394032:85 , srv53631372:85 , srv984223255:85 , srv108110810:86 , srv1175936748:86 , srv1454271534:86 , srv145800934:86 , srv1540499464:86 , srv1576350282:86 , srv1624135761:86 , srv1743720058:86 , srv1838823333:86 , srv2002688569:86 , srv2032896893:86 , srv2051197484:86 , srv2115559353:86 , srv281713930:86 , srv303296620:86 , srv303575484:86 , srv913854871:86 , srv1088020356:87 , srv1113767206:87 , srv1391716564:87 , srv1426608652:87 , srv1427287819:87 , srv1725466141:87 , srv1826565413:87 , srv1853175199:87 , srv1952642552:87 , srv1957385070:87 , srv2141102789:87 , srv33699245:87 , srv344151307:87 , srv70718470:87 , srv843820921:87 , srv964924983:87 , srv975311096:87 , srv1132738059:88 , srv1274385078:88 , srv1609860167:88 , srv1767870374:88 , srv1782743105:88 , srv1970025774:88 , srv585387119:88 , srv635831529:88 , srv95633208:88 , srv976159865:88 , srv1255692040:89 , srv1721279792:89 , srv1884620558:89 , srv1899056738:89 , srv2063310880:89 , srv2070327025:89 , srv382834144:89 , srv62431695:89 , srv748748971:89 , srv1125581439:90 , srv1996957541:90 , srv1997401603:90 , srv1280376766:91 , srv1869974786:91 , srv1753123535:92 , srv477929048:95 , srv1349459687:96 , srv1423194977:97 , srv84159017:443 } 2023-04-29 21:53:34,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:34,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1872427821=41, srv1255692040=9, srv453517217=73, srv65161275=82, srv585387119=79, srv635831529=81, srv290239679=63, srv1175936748=8, srv1884620558=43, srv279865199=61, srv1113767206=4, srv1996957541=51, srv843820921=89, srv802363243=87, srv382834144=70, srv1423194977=16, srv95633208=92, srv2141102789=60, srv964924983=93, srv532835385=77, srv344151307=68, srv1454271534=19, srv2002688569=53, srv45139609=72, srv477929048=76, srv974591432=96, srv1349459687=12, srv728275827=85, srv1074747057=1, srv84159017=88, srv1414036783=15, srv1767870374=33, srv1782743105=34, srv1274385078=10, srv1624135761=26, srv662436256=83, srv1843076639=38, srv1952642552=46, srv1737490397=30, srv53631372=78, srv976159865=98, srv1576350282=24, srv1128653748=6, srv1869974786=40, srv1899056738=44, srv1535359165=22, srv349234357=69, srv1876934701=42, srv2032896893=54, srv303296620=65, srv974517645=95, srv1964127812=48, srv469732788=74, srv2063310880=57, srv1975796237=50, srv984223255=99, srv1753123535=32, srv1721279792=28, srv1484135669=21, srv2115559353=59, srv1970025774=49, srv748748971=86, srv180895636=35, srv1540499464=23, srv294328551=64, srv1125581439=5, srv70718470=84, srv966008921=94, srv303575484=66, srv1280376766=11, srv1088020356=3, srv281713930=62, srv1427287819=18, srv62431695=80, srv1043364553=0, srv1714356353=27, srv1826565413=36, srv1838823333=37, srv1426608652=17, srv2070327025=58, srv145800934=20, srv431065617=71, srv2051197484=56, srv1725466141=29, srv975311096=97, srv1132738059=7, srv1943779189=45, srv2036993192=55, srv913854871=90, srv1609860167=25, srv1997401603=52, srv1391716564=14, srv915605857=91, srv1387724456=13, srv1743720058=31, srv1957385070=47, srv33699245=67, srv472394032=75, srv1853175199=39, srv108110810=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:34,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:34,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:34,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:34,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:34,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-29 21:53:34,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-29 21:53:34,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-29 21:53:34,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-29 21:53:34,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-29 21:53:34,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-29 21:53:34,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-29 21:53:34,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-29 21:53:34,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-29 21:53:34,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-29 21:53:34,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-29 21:53:34,572 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03974196608514426, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056443410473310224, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03974196608514426, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17633333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24191666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:34,573 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-29 21:53:34,573 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18097341175123755, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03974196608514426, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056443410473310224, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03974196608514426, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17633333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24191666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-29 21:53:34,897 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 2640 regions; Going from a computed imbalance of 0.18097341175123755 to a new imbalance of 0.014259894539022528. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02161343427626936); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02161343427626936); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:34,909 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1423194977:72 , srv1349459687:75 , srv477929048:76 , srv1753123535:79 , srv1996957541:80 , srv1088020356:81 , srv1125581439:81 , srv1280376766:81 , srv1869974786:81 , srv1884620558:81 , srv2070327025:81 , srv1609860167:82 , srv1255692040:83 , srv1427287819:83 , srv1576350282:83 , srv1721279792:83 , srv1767870374:83 , srv1782743105:83 , srv2063310880:83 , srv382834144:83 , srv62431695:83 , srv1274385078:84 , srv1391716564:84 , srv1899056738:84 , srv1952642552:84 , srv1970025774:84 , srv1997401603:84 , srv2032896893:84 , srv585387119:84 , srv748748971:84 , srv95633208:84 , srv1113767206:85 , srv1132738059:85 , srv1426608652:85 , srv145800934:85 , srv1725466141:85 , srv1826565413:85 , srv1876934701:85 , srv1943779189:85 , srv2141102789:85 , srv303296620:85 , srv33699245:85 , srv70718470:85 , srv964924983:85 , srv974517645:85 , srv975311096:85 , srv976159865:85 , srv1743720058:86 , srv1957385070:86 , srv1964127812:86 , srv2002688569:86 , srv303575484:86 , srv344151307:86 , srv472394032:86 , srv843820921:86 , srv1043364553:87 , srv1175936748:87 , srv1414036783:87 , srv1714356353:87 , srv1838823333:87 , srv1853175199:87 , srv2051197484:87 , srv281713930:87 , srv294328551:87 , srv431065617:87 , srv453517217:87 , srv53631372:87 , srv635831529:87 , srv915605857:87 , srv984223255:87 , srv1074747057:88 , srv108110810:88 , srv1484135669:88 , srv1737490397:88 , srv1975796237:88 , srv2115559353:88 , srv469732788:88 , srv532835385:88 , srv913854871:88 , srv974591432:88 , srv1128653748:89 , srv1387724456:89 , srv1454271534:89 , srv1535359165:89 , srv1540499464:89 , srv1872427821:89 , srv349234357:89 , srv45139609:89 , srv802363243:89 , srv180895636:90 , srv1843076639:90 , srv279865199:90 , srv290239679:90 , srv65161275:90 , srv728275827:90 , srv966008921:90 , srv1624135761:91 , srv662436256:92 , srv2036993192:93 , srv84159017:523 } 2023-04-29 21:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1872427821=41, srv1255692040=9, srv453517217=73, srv65161275=82, srv585387119=79, srv635831529=81, srv290239679=63, srv1175936748=8, srv1884620558=43, srv279865199=61, srv1113767206=4, srv1996957541=51, srv843820921=89, srv802363243=87, srv382834144=70, srv1423194977=16, srv95633208=92, srv2141102789=60, srv964924983=93, srv532835385=77, srv344151307=68, srv1454271534=19, srv2002688569=53, srv45139609=72, srv477929048=76, srv974591432=96, srv1349459687=12, srv728275827=85, srv1074747057=1, srv84159017=88, srv1414036783=15, srv1767870374=33, srv1782743105=34, srv1274385078=10, srv1624135761=26, srv662436256=83, srv1843076639=38, srv1952642552=46, srv1737490397=30, srv53631372=78, srv976159865=98, srv1576350282=24, srv1128653748=6, srv1869974786=40, srv1899056738=44, srv1535359165=22, srv349234357=69, srv1876934701=42, srv2032896893=54, srv303296620=65, srv974517645=95, srv1964127812=48, srv469732788=74, srv2063310880=57, srv1975796237=50, srv984223255=99, srv1753123535=32, srv1721279792=28, srv1484135669=21, srv2115559353=59, srv1970025774=49, srv748748971=86, srv180895636=35, srv1540499464=23, srv294328551=64, srv1125581439=5, srv70718470=84, srv966008921=94, srv303575484=66, srv1280376766=11, srv1088020356=3, srv281713930=62, srv1427287819=18, srv62431695=80, srv1043364553=0, srv1714356353=27, srv1826565413=36, srv1838823333=37, srv1426608652=17, srv2070327025=58, srv145800934=20, srv431065617=71, srv2051197484=56, srv1725466141=29, srv975311096=97, srv1132738059=7, srv1943779189=45, srv2036993192=55, srv913854871=90, srv1609860167=25, srv1997401603=52, srv1391716564=14, srv915605857=91, srv1387724456=13, srv1743720058=31, srv1957385070=47, srv33699245=67, srv472394032=75, srv1853175199=39, srv108110810=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:34,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:34,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:34,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:34,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:34,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:34,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:34,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:34,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-29 21:53:34,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-29 21:53:34,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-29 21:53:34,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-29 21:53:34,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-29 21:53:34,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-29 21:53:34,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-29 21:53:34,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-29 21:53:34,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-29 21:53:34,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-29 21:53:34,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-29 21:53:35,110 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02161343427626936); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02161343427626936); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:35,110 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-29 21:53:35,111 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014241408863920273, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02161343427626936); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02161343427626936); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-29 21:53:35,417 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 428 ms to try 5000 different iterations. Found a solution that moves 989 regions; Going from a computed imbalance of 0.014241408863920273 to a new imbalance of 9.078242978953793E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071852198964762574); MoveCostFunction : (multiplier=7.0, imbalance=0.10988888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:35,420 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1624135761:80 , srv662436256:81 , srv1454271534:82 , srv635831529:82 , srv108110810:83 , srv1540499464:83 , srv1843076639:83 , srv1997401603:83 , srv2036993192:83 , srv2115559353:83 , srv65161275:83 , srv1128653748:84 , srv1484135669:84 , srv1535359165:84 , srv1899056738:84 , srv344151307:84 , srv802363243:84 , srv913854871:84 , srv976159865:84 , srv1074747057:85 , srv1113767206:85 , srv1132738059:85 , srv1175936748:85 , srv1255692040:85 , srv1274385078:85 , srv1387724456:85 , srv1414036783:85 , srv1426608652:85 , srv1721279792:85 , srv1725466141:85 , srv1737490397:85 , srv1743720058:85 , srv1838823333:85 , srv1853175199:85 , srv1957385070:85 , srv2051197484:85 , srv2063310880:85 , srv279865199:85 , srv281713930:85 , srv294328551:85 , srv33699245:85 , srv349234357:85 , srv45139609:85 , srv453517217:85 , srv469732788:85 , srv53631372:85 , srv748748971:85 , srv843820921:85 , srv95633208:85 , srv966008921:85 , srv984223255:85 , srv1043364553:86 , srv1125581439:86 , srv1280376766:86 , srv1391716564:86 , srv145800934:86 , srv1753123535:86 , srv1767870374:86 , srv180895636:86 , srv1869974786:86 , srv1952642552:86 , srv1964127812:86 , srv1970025774:86 , srv1975796237:86 , srv1996957541:86 , srv2002688569:86 , srv2141102789:86 , srv290239679:86 , srv303296620:86 , srv382834144:86 , srv431065617:86 , srv477929048:86 , srv585387119:86 , srv62431695:86 , srv70718470:86 , srv728275827:86 , srv915605857:86 , srv964924983:86 , srv974591432:86 , srv975311096:86 , srv1349459687:87 , srv1782743105:87 , srv1826565413:87 , srv1872427821:87 , srv1876934701:87 , srv1943779189:87 , srv2070327025:87 , srv303575484:87 , srv472394032:87 , srv532835385:87 , srv1423194977:88 , srv1427287819:88 , srv1576350282:88 , srv1609860167:88 , srv1714356353:88 , srv1884620558:88 , srv2032896893:88 , srv1088020356:89 , srv974517645:89 , srv84159017:544 } 2023-04-29 21:53:35,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:35,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1872427821=41, srv1255692040=9, srv453517217=73, srv65161275=82, srv585387119=79, srv635831529=81, srv290239679=63, srv1175936748=8, srv1884620558=43, srv279865199=61, srv1113767206=4, srv1996957541=51, srv843820921=89, srv802363243=87, srv382834144=70, srv1423194977=16, srv95633208=92, srv2141102789=60, srv964924983=93, srv532835385=77, srv344151307=68, srv1454271534=19, srv2002688569=53, srv45139609=72, srv477929048=76, srv974591432=96, srv1349459687=12, srv728275827=85, srv1074747057=1, srv84159017=88, srv1414036783=15, srv1767870374=33, srv1782743105=34, srv1274385078=10, srv1624135761=26, srv662436256=83, srv1843076639=38, srv1952642552=46, srv1737490397=30, srv53631372=78, srv976159865=98, srv1576350282=24, srv1128653748=6, srv1869974786=40, srv1899056738=44, srv1535359165=22, srv349234357=69, srv1876934701=42, srv2032896893=54, srv303296620=65, srv974517645=95, srv1964127812=48, srv469732788=74, srv2063310880=57, srv1975796237=50, srv984223255=99, srv1753123535=32, srv1721279792=28, srv1484135669=21, srv2115559353=59, srv1970025774=49, srv748748971=86, srv180895636=35, srv1540499464=23, srv294328551=64, srv1125581439=5, srv70718470=84, srv966008921=94, srv303575484=66, srv1280376766=11, srv1088020356=3, srv281713930=62, srv1427287819=18, srv62431695=80, srv1043364553=0, srv1714356353=27, srv1826565413=36, srv1838823333=37, srv1426608652=17, srv2070327025=58, srv145800934=20, srv431065617=71, srv2051197484=56, srv1725466141=29, srv975311096=97, srv1132738059=7, srv1943779189=45, srv2036993192=55, srv913854871=90, srv1609860167=25, srv1997401603=52, srv1391716564=14, srv915605857=91, srv1387724456=13, srv1743720058=31, srv1957385070=47, srv33699245=67, srv472394032=75, srv1853175199=39, srv108110810=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:35,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:35,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-29 21:53:35,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-29 21:53:35,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-29 21:53:35,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-29 21:53:35,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-29 21:53:35,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-29 21:53:35,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-29 21:53:35,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-29 21:53:35,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-29 21:53:35,566 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071852198964762574); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:35,566 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-29 21:53:35,566 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.00899171881693E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071852198964762574); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-29 21:53:35,961 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations. Found a solution that moves 289 regions; Going from a computed imbalance of 9.00899171881693E-4 to a new imbalance of 2.8952753629202826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.03211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:35,962 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1609860167:83 , srv1714356353:83 , srv1826565413:83 , srv1853175199:83 , srv303575484:83 , srv70718470:83 , srv728275827:83 , srv748748971:83 , srv1043364553:84 , srv1088020356:84 , srv1128653748:84 , srv1175936748:84 , srv1274385078:84 , srv1280376766:84 , srv1349459687:84 , srv1423194977:84 , srv1427287819:84 , srv1540499464:84 , srv1737490397:84 , srv1753123535:84 , srv1782743105:84 , srv1838823333:84 , srv1843076639:84 , srv1869974786:84 , srv1884620558:84 , srv1970025774:84 , srv1975796237:84 , srv2002688569:84 , srv2032896893:84 , srv2141102789:84 , srv281713930:84 , srv294328551:84 , srv382834144:84 , srv431065617:84 , srv472394032:84 , srv532835385:84 , srv585387119:84 , srv62431695:84 , srv843820921:84 , srv913854871:84 , srv964924983:84 , srv974517645:84 , srv975311096:84 , srv1074747057:85 , srv108110810:85 , srv1113767206:85 , srv1125581439:85 , srv1132738059:85 , srv1255692040:85 , srv1387724456:85 , srv1391716564:85 , srv1414036783:85 , srv1426608652:85 , srv1454271534:85 , srv145800934:85 , srv1484135669:85 , srv1576350282:85 , srv1624135761:85 , srv1721279792:85 , srv1725466141:85 , srv1743720058:85 , srv1767870374:85 , srv180895636:85 , srv1872427821:85 , srv1876934701:85 , srv1899056738:85 , srv1943779189:85 , srv1957385070:85 , srv1964127812:85 , srv1997401603:85 , srv2036993192:85 , srv2051197484:85 , srv2063310880:85 , srv2070327025:85 , srv2115559353:85 , srv279865199:85 , srv290239679:85 , srv303296620:85 , srv33699245:85 , srv344151307:85 , srv349234357:85 , srv45139609:85 , srv453517217:85 , srv469732788:85 , srv477929048:85 , srv53631372:85 , srv635831529:85 , srv65161275:85 , srv662436256:85 , srv802363243:85 , srv915605857:85 , srv95633208:85 , srv974591432:85 , srv976159865:85 , srv984223255:85 , srv1535359165:86 , srv1952642552:86 , srv1996957541:86 , srv966008921:86 , srv84159017:632 } 2023-04-29 21:53:35,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:35,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1872427821=41, srv1255692040=9, srv453517217=73, srv65161275=82, srv585387119=79, srv635831529=81, srv290239679=63, srv1175936748=8, srv1884620558=43, srv279865199=61, srv1113767206=4, srv1996957541=51, srv843820921=89, srv802363243=87, srv382834144=70, srv1423194977=16, srv95633208=92, srv2141102789=60, srv964924983=93, srv532835385=77, srv344151307=68, srv1454271534=19, srv2002688569=53, srv45139609=72, srv477929048=76, srv974591432=96, srv1349459687=12, srv728275827=85, srv1074747057=1, srv84159017=88, srv1414036783=15, srv1767870374=33, srv1782743105=34, srv1274385078=10, srv1624135761=26, srv662436256=83, srv1843076639=38, srv1952642552=46, srv1737490397=30, srv53631372=78, srv976159865=98, srv1576350282=24, srv1128653748=6, srv1869974786=40, srv1899056738=44, srv1535359165=22, srv349234357=69, srv1876934701=42, srv2032896893=54, srv303296620=65, srv974517645=95, srv1964127812=48, srv469732788=74, srv2063310880=57, srv1975796237=50, srv984223255=99, srv1753123535=32, srv1721279792=28, srv1484135669=21, srv2115559353=59, srv1970025774=49, srv748748971=86, srv180895636=35, srv1540499464=23, srv294328551=64, srv1125581439=5, srv70718470=84, srv966008921=94, srv303575484=66, srv1280376766=11, srv1088020356=3, srv281713930=62, srv1427287819=18, srv62431695=80, srv1043364553=0, srv1714356353=27, srv1826565413=36, srv1838823333=37, srv1426608652=17, srv2070327025=58, srv145800934=20, srv431065617=71, srv2051197484=56, srv1725466141=29, srv975311096=97, srv1132738059=7, srv1943779189=45, srv2036993192=55, srv913854871=90, srv1609860167=25, srv1997401603=52, srv1391716564=14, srv915605857=91, srv1387724456=13, srv1743720058=31, srv1957385070=47, srv33699245=67, srv472394032=75, srv1853175199=39, srv108110810=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:35,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:35,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:35,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:35,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-29 21:53:35,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-29 21:53:35,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-29 21:53:35,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-29 21:53:35,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-29 21:53:35,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-29 21:53:35,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-29 21:53:35,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-29 21:53:35,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-29 21:53:35,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-29 21:53:35,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-29 21:53:35,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-29 21:53:35,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-29 21:53:36,122 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:36,122 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-29 21:53:36,123 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6929132377478546E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-29 21:53:36,362 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 2.6929132377478546E-5 to a new imbalance of 7.885481990829257E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:36,363 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1132738059:83 , srv1387724456:83 , srv1535359165:83 , srv1952642552:83 , srv1957385070:83 , srv2051197484:83 , srv349234357:83 , srv966008921:83 , srv1043364553:84 , srv1074747057:84 , srv108110810:84 , srv1113767206:84 , srv1125581439:84 , srv1128653748:84 , srv1175936748:84 , srv1255692040:84 , srv1280376766:84 , srv1349459687:84 , srv1391716564:84 , srv1414036783:84 , srv1426608652:84 , srv1427287819:84 , srv1454271534:84 , srv145800934:84 , srv1484135669:84 , srv1540499464:84 , srv1576350282:84 , srv1624135761:84 , srv1714356353:84 , srv1721279792:84 , srv1725466141:84 , srv1737490397:84 , srv1743720058:84 , srv1767870374:84 , srv1782743105:84 , srv180895636:84 , srv1826565413:84 , srv1838823333:84 , srv1843076639:84 , srv1853175199:84 , srv1869974786:84 , srv1872427821:84 , srv1876934701:84 , srv1884620558:84 , srv1899056738:84 , srv1943779189:84 , srv1964127812:84 , srv1970025774:84 , srv1975796237:84 , srv1996957541:84 , srv1997401603:84 , srv2002688569:84 , srv2032896893:84 , srv2036993192:84 , srv2063310880:84 , srv2070327025:84 , srv2115559353:84 , srv2141102789:84 , srv279865199:84 , srv281713930:84 , srv290239679:84 , srv303296620:84 , srv303575484:84 , srv33699245:84 , srv344151307:84 , srv382834144:84 , srv431065617:84 , srv45139609:84 , srv453517217:84 , srv469732788:84 , srv472394032:84 , srv477929048:84 , srv532835385:84 , srv53631372:84 , srv585387119:84 , srv62431695:84 , srv635831529:84 , srv65161275:84 , srv662436256:84 , srv728275827:84 , srv802363243:84 , srv843820921:84 , srv913854871:84 , srv915605857:84 , srv95633208:84 , srv964924983:84 , srv974517645:84 , srv974591432:84 , srv975311096:84 , srv976159865:84 , srv984223255:84 , srv1088020356:85 , srv1274385078:85 , srv1423194977:85 , srv1609860167:85 , srv1753123535:85 , srv294328551:85 , srv70718470:85 , srv748748971:85 , srv84159017:684 } 2023-04-29 21:53:36,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:36,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1872427821=41, srv1255692040=9, srv453517217=73, srv65161275=82, srv585387119=79, srv635831529=81, srv290239679=63, srv1175936748=8, srv1884620558=43, srv279865199=61, srv1113767206=4, srv1996957541=51, srv843820921=89, srv802363243=87, srv382834144=70, srv1423194977=16, srv95633208=92, srv2141102789=60, srv964924983=93, srv532835385=77, srv344151307=68, srv1454271534=19, srv2002688569=53, srv45139609=72, srv477929048=76, srv974591432=96, srv1349459687=12, srv728275827=85, srv1074747057=1, srv84159017=88, srv1414036783=15, srv1767870374=33, srv1782743105=34, srv1274385078=10, srv1624135761=26, srv662436256=83, srv1843076639=38, srv1952642552=46, srv1737490397=30, srv53631372=78, srv976159865=98, srv1576350282=24, srv1128653748=6, srv1869974786=40, srv1899056738=44, srv1535359165=22, srv349234357=69, srv1876934701=42, srv2032896893=54, srv303296620=65, srv974517645=95, srv1964127812=48, srv469732788=74, srv2063310880=57, srv1975796237=50, srv984223255=99, srv1753123535=32, srv1721279792=28, srv1484135669=21, srv2115559353=59, srv1970025774=49, srv748748971=86, srv180895636=35, srv1540499464=23, srv294328551=64, srv1125581439=5, srv70718470=84, srv966008921=94, srv303575484=66, srv1280376766=11, srv1088020356=3, srv281713930=62, srv1427287819=18, srv62431695=80, srv1043364553=0, srv1714356353=27, srv1826565413=36, srv1838823333=37, srv1426608652=17, srv2070327025=58, srv145800934=20, srv431065617=71, srv2051197484=56, srv1725466141=29, srv975311096=97, srv1132738059=7, srv1943779189=45, srv2036993192=55, srv913854871=90, srv1609860167=25, srv1997401603=52, srv1391716564=14, srv915605857=91, srv1387724456=13, srv1743720058=31, srv1957385070=47, srv33699245=67, srv472394032=75, srv1853175199=39, srv108110810=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:36,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:36,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:36,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:36,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-29 21:53:36,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-29 21:53:36,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-29 21:53:36,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-29 21:53:36,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-29 21:53:36,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-29 21:53:36,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-29 21:53:36,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-29 21:53:36,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-29 21:53:36,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-29 21:53:36,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-29 21:53:36,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-29 21:53:36,515 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:36,516 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1132738059:83 , srv1387724456:83 , srv1535359165:83 , srv1952642552:83 , srv1957385070:83 , srv2051197484:83 , srv349234357:83 , srv966008921:83 , srv1043364553:84 , srv1074747057:84 , srv108110810:84 , srv1113767206:84 , srv1125581439:84 , srv1128653748:84 , srv1175936748:84 , srv1255692040:84 , srv1280376766:84 , srv1349459687:84 , srv1391716564:84 , srv1414036783:84 , srv1426608652:84 , srv1427287819:84 , srv1454271534:84 , srv145800934:84 , srv1484135669:84 , srv1540499464:84 , srv1576350282:84 , srv1624135761:84 , srv1714356353:84 , srv1721279792:84 , srv1725466141:84 , srv1737490397:84 , srv1743720058:84 , srv1767870374:84 , srv1782743105:84 , srv180895636:84 , srv1826565413:84 , srv1838823333:84 , srv1843076639:84 , srv1853175199:84 , srv1869974786:84 , srv1872427821:84 , srv1876934701:84 , srv1884620558:84 , srv1899056738:84 , srv1943779189:84 , srv1964127812:84 , srv1970025774:84 , srv1975796237:84 , srv1996957541:84 , srv1997401603:84 , srv2002688569:84 , srv2032896893:84 , srv2036993192:84 , srv2063310880:84 , srv2070327025:84 , srv2115559353:84 , srv2141102789:84 , srv279865199:84 , srv281713930:84 , srv290239679:84 , srv303296620:84 , srv303575484:84 , srv33699245:84 , srv344151307:84 , srv382834144:84 , srv431065617:84 , srv45139609:84 , srv453517217:84 , srv469732788:84 , srv472394032:84 , srv477929048:84 , srv532835385:84 , srv53631372:84 , srv585387119:84 , srv62431695:84 , srv635831529:84 , srv65161275:84 , srv662436256:84 , srv728275827:84 , srv802363243:84 , srv843820921:84 , srv913854871:84 , srv915605857:84 , srv95633208:84 , srv964924983:84 , srv974517645:84 , srv974591432:84 , srv975311096:84 , srv976159865:84 , srv984223255:84 , srv1088020356:85 , srv1274385078:85 , srv1423194977:85 , srv1609860167:85 , srv1753123535:85 , srv294328551:85 , srv70718470:85 , srv748748971:85 , srv84159017:684 } 2023-04-29 21:53:36,552 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=335 (was 321) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 171), AvailableMemoryMB=10541 (was 11422) 2023-04-29 21:53:36,560 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=335, ProcessCount=170, AvailableMemoryMB=10541 2023-04-29 21:53:36,561 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-29 21:53:36,561 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-29 21:53:36,573 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004123462:84 , srv1084208400:84 , srv1547452835:84 , srv1793742499:84 , srv1800908768:84 , srv1803922471:84 , srv1835209746:84 , srv1838267406:84 , srv185400158:84 , srv1892559652:84 , srv1895569940:84 , srv1896400746:84 , srv1989050453:84 , srv2009138779:84 , srv2109302363:84 , srv2145802110:84 , srv225740433:84 , srv386775739:84 , srv388129576:84 , srv449629964:84 , srv597444833:84 , srv610151107:84 , srv791689801:84 , srv807396169:84 , srv815512692:84 , srv88680793:84 , srv893014883:84 , srv904327637:84 , srv940484329:84 , srv1343938450:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-29 21:53:36,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:53:36,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv388129576=19, srv1793742499=4, srv2145802110=16, srv815512692=25, srv1547452835=3, srv2109302363=15, srv185400158=9, srv1803922471=6, srv1895569940=11, srv1084208400=1, srv225740433=17, srv893014883=27, srv2009138779=14, srv904327637=28, srv1343938450=2, srv1896400746=12, srv597444833=21, srv449629964=20, srv1989050453=13, srv610151107=22, srv1835209746=7, srv940484329=29, srv791689801=23, srv807396169=24, srv1892559652=10, srv1838267406=8, srv1004123462=0, srv386775739=18, srv88680793=26, srv1800908768=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:53:36,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:53:36,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:53:36,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:53:36,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:53:36,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:53:36,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:53:36,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:53:36,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-29 21:53:36,619 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1080009774226813, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:53:36,619 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912533350817583, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1080009774226813, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-29 21:55:36,595 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6961421 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912533350817583 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:55:36,602 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004123462:90 , srv1084208400:90 , srv1343938450:90 , srv1547452835:90 , srv1793742499:90 , srv1800908768:90 , srv1803922471:90 , srv1835209746:90 , srv1838267406:90 , srv185400158:90 , srv1892559652:90 , srv1895569940:90 , srv1896400746:90 , srv1989050453:90 , srv2009138779:90 , srv2109302363:90 , srv2145802110:90 , srv225740433:90 , srv386775739:90 , srv388129576:90 , srv449629964:90 , srv597444833:90 , srv610151107:90 , srv791689801:90 , srv807396169:90 , srv815512692:90 , srv88680793:90 , srv893014883:90 , srv904327637:90 , srv940484329:90 } 2023-04-29 21:55:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-29 21:55:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv388129576=19, srv1793742499=4, srv2145802110=16, srv815512692=25, srv1547452835=3, srv2109302363=15, srv185400158=9, srv1803922471=6, srv1895569940=11, srv1084208400=1, srv225740433=17, srv893014883=27, srv2009138779=14, srv904327637=28, srv1343938450=2, srv1896400746=12, srv597444833=21, srv449629964=20, srv1989050453=13, srv610151107=22, srv1835209746=7, srv940484329=29, srv791689801=23, srv807396169=24, srv1892559652=10, srv1838267406=8, srv1004123462=0, srv386775739=18, srv88680793=26, srv1800908768=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-29 21:55:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-29 21:55:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-29 21:55:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-29 21:55:36,627 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-29 21:55:36,627 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1004123462:90 , srv1084208400:90 , srv1343938450:90 , srv1547452835:90 , srv1793742499:90 , srv1800908768:90 , srv1803922471:90 , srv1835209746:90 , srv1838267406:90 , srv185400158:90 , srv1892559652:90 , srv1895569940:90 , srv1896400746:90 , srv1989050453:90 , srv2009138779:90 , srv2109302363:90 , srv2145802110:90 , srv225740433:90 , srv386775739:90 , srv388129576:90 , srv449629964:90 , srv597444833:90 , srv610151107:90 , srv791689801:90 , srv807396169:90 , srv815512692:90 , srv88680793:90 , srv893014883:90 , srv904327637:90 , srv940484329:90 } 2023-04-29 21:55:36,641 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 335), ProcessCount=170 (was 170), AvailableMemoryMB=10515 (was 10541)