2023-12-27 16:53:20,469 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-27 16:53:20,803 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-27 16:53:20,923 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-27 16:53:20,992 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-27 16:53:21,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-27 16:53:21,019 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=302, ProcessCount=172, AvailableMemoryMB=11538 2023-12-27 16:53:21,022 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-27 16:53:21,023 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-27 16:53:21,156 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001647201:84 , srv1009767501:84 , srv1016985188:84 , srv1040104660:84 , srv1076899667:84 , srv1092723356:84 , srv109466082:84 , srv1097727376:84 , srv1114200426:84 , srv1123811656:84 , srv1124415620:84 , srv116866562:84 , srv1205422369:84 , srv1230874047:84 , srv1263841632:84 , srv1267412393:84 , srv1270609625:84 , srv1298036119:84 , srv1315261174:84 , srv1325441163:84 , srv1351882054:84 , srv1359459860:84 , srv1397352207:84 , srv1432689957:84 , srv1468239694:84 , srv1506518013:84 , srv1523662010:84 , srv1560411052:84 , srv1573126311:84 , srv1601093463:84 , srv1624909744:84 , srv1648373453:84 , srv1649642589:84 , srv1669708836:84 , srv1709088926:84 , srv1726782423:84 , srv1746176898:84 , srv1790788064:84 , srv1808687888:84 , srv1812756772:84 , srv1828715867:84 , srv1874001807:84 , srv1900227448:84 , srv1909135405:84 , srv1962039392:84 , srv1990843597:84 , srv1992252086:84 , srv2004586079:84 , srv2014865861:84 , srv2034518343:84 , srv2057520330:84 , srv2068256546:84 , srv2072788877:84 , srv2080280625:84 , srv2095528547:84 , srv212235009:84 , srv2144146546:84 , srv247567755:84 , srv261284709:84 , srv27400749:84 , srv292505542:84 , srv311745780:84 , srv314193770:84 , srv381697402:84 , srv384570389:84 , srv408177529:84 , srv410088693:84 , srv424806870:84 , srv447178306:84 , srv454393975:84 , srv454432571:84 , srv459531023:84 , srv48981370:84 , srv525107640:84 , srv525167318:84 , srv574423986:84 , srv578984282:84 , srv580915449:84 , srv590960780:84 , srv594974274:84 , srv612537108:84 , srv636444712:84 , srv696702450:84 , srv702320782:84 , srv721402728:84 , srv72848611:84 , srv749714530:84 , srv779786899:84 , srv827309:84 , srv838479219:84 , srv84439471:84 , srv894568010:84 , srv897095270:84 , srv900410742:84 , srv906449752:84 , srv946573301:84 , srv995373230:84 , srv995563620:84 , srv997803663:84 , srv1115327261:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-27 16:53:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:21,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2034518343=50, srv1909135405=44, srv2080280625=54, srv212235009=56, srv594974274=80, srv696702450=83, srv1709088926=35, srv578984282=77, srv48981370=73, srv1397352207=23, srv1076899667=4, srv1092723356=5, srv827309=89, srv1115327261=9, srv27400749=60, srv1648373453=32, srv1573126311=29, srv1992252086=47, srv1351882054=21, srv995373230=97, srv1506518013=26, srv525167318=75, srv1790788064=38, srv1468239694=25, srv2068256546=52, srv1263841632=15, srv1267412393=16, srv900410742=94, srv1560411052=28, srv261284709=59, srv838479219=90, srv1601093463=30, srv2095528547=55, srv408177529=66, srv995563620=98, srv384570389=65, srv897095270=93, srv1990843597=46, srv721402728=85, srv906449752=95, srv410088693=67, srv459531023=72, srv1124415620=11, srv247567755=58, srv1123811656=10, srv1624909744=31, srv1270609625=17, srv1432689957=24, srv84439471=91, srv424806870=68, srv702320782=84, srv749714530=87, srv1315261174=19, srv292505542=61, srv2004586079=48, srv1016985188=2, srv1649642589=33, srv1205422369=13, srv590960780=79, srv779786899=88, srv1808687888=39, srv1097727376=7, srv447178306=69, srv1669708836=34, srv2072788877=53, srv1812756772=40, srv1359459860=22, srv1962039392=45, srv1040104660=3, srv636444712=82, srv454393975=70, srv311745780=62, srv454432571=71, srv1726782423=36, srv314193770=63, srv1114200426=8, srv580915449=78, srv997803663=99, srv946573301=96, srv2057520330=51, srv1298036119=18, srv1828715867=41, srv1746176898=37, srv1523662010=27, srv894568010=92, srv1874001807=42, srv612537108=81, srv1001647201=0, srv116866562=12, srv1325441163=20, srv381697402=64, srv525107640=74, srv2144146546=57, srv1009767501=1, srv72848611=86, srv1230874047=14, srv2014865861=49, srv109466082=6, srv1900227448=43, srv574423986=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:21,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:21,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:21,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:21,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-27 16:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-27 16:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-27 16:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-27 16:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-27 16:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-27 16:53:21,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-27 16:53:21,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-27 16:53:21,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-27 16:53:21,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-27 16:53:21,603 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:21,603 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-27 16:53:21,603 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-27 16:53:22,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 750 ms to try 5000 different iterations. Found a solution that moves 4703 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1768613306334819. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0389095664123722, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05650303171151609, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5225555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0389095664123722, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:22,092 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1601093463:82 , srv1040104660:83 , srv1123811656:83 , srv1523662010:83 , srv1808687888:83 , srv1962039392:83 , srv212235009:83 , srv1016985188:84 , srv1092723356:84 , srv1263841632:84 , srv1746176898:84 , srv1790788064:84 , srv311745780:84 , srv381697402:84 , srv525107640:84 , srv574423986:84 , srv578984282:84 , srv590960780:84 , srv897095270:84 , srv900410742:84 , srv1097727376:85 , srv1325441163:85 , srv1812756772:85 , srv1874001807:85 , srv2068256546:85 , srv292505542:85 , srv314193770:85 , srv424806870:85 , srv454393975:85 , srv459531023:85 , srv721402728:85 , srv838479219:85 , srv997803663:85 , srv109466082:86 , srv1267412393:86 , srv1315261174:86 , srv1624909744:86 , srv1828715867:86 , srv1900227448:86 , srv2014865861:86 , srv2034518343:86 , srv384570389:86 , srv408177529:86 , srv580915449:86 , srv72848611:86 , srv779786899:86 , srv894568010:86 , srv995563620:86 , srv1076899667:87 , srv116866562:87 , srv1205422369:87 , srv1230874047:87 , srv1270609625:87 , srv1298036119:87 , srv1359459860:87 , srv1397352207:87 , srv1468239694:87 , srv1560411052:87 , srv1648373453:87 , srv1669708836:87 , srv1726782423:87 , srv1990843597:87 , srv2004586079:87 , srv2057520330:87 , srv247567755:87 , srv27400749:87 , srv410088693:87 , srv525167318:87 , srv696702450:87 , srv827309:87 , srv84439471:87 , srv906449752:87 , srv995373230:87 , srv1009767501:88 , srv1351882054:88 , srv1506518013:88 , srv1573126311:88 , srv1649642589:88 , srv1909135405:88 , srv1992252086:88 , srv2072788877:88 , srv594974274:88 , srv1001647201:89 , srv1114200426:89 , srv1124415620:89 , srv1709088926:89 , srv261284709:89 , srv447178306:89 , srv454432571:89 , srv612537108:89 , srv636444712:89 , srv2080280625:90 , srv2095528547:90 , srv2144146546:90 , srv702320782:90 , srv1432689957:91 , srv749714530:91 , srv946573301:93 , srv48981370:94 , srv1115327261:436 } 2023-12-27 16:53:22,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2034518343=50, srv1909135405=44, srv2080280625=54, srv212235009=56, srv594974274=80, srv696702450=83, srv1709088926=35, srv578984282=77, srv48981370=73, srv1397352207=23, srv1076899667=4, srv1092723356=5, srv827309=89, srv1115327261=9, srv27400749=60, srv1648373453=32, srv1573126311=29, srv1992252086=47, srv1351882054=21, srv995373230=97, srv1506518013=26, srv525167318=75, srv1790788064=38, srv1468239694=25, srv2068256546=52, srv1263841632=15, srv1267412393=16, srv900410742=94, srv1560411052=28, srv261284709=59, srv838479219=90, srv1601093463=30, srv2095528547=55, srv408177529=66, srv995563620=98, srv384570389=65, srv897095270=93, srv1990843597=46, srv721402728=85, srv906449752=95, srv410088693=67, srv459531023=72, srv1124415620=11, srv247567755=58, srv1123811656=10, srv1624909744=31, srv1270609625=17, srv1432689957=24, srv84439471=91, srv424806870=68, srv702320782=84, srv749714530=87, srv1315261174=19, srv292505542=61, srv2004586079=48, srv1016985188=2, srv1649642589=33, srv1205422369=13, srv590960780=79, srv779786899=88, srv1808687888=39, srv1097727376=7, srv447178306=69, srv1669708836=34, srv2072788877=53, srv1812756772=40, srv1359459860=22, srv1962039392=45, srv1040104660=3, srv636444712=82, srv454393975=70, srv311745780=62, srv454432571=71, srv1726782423=36, srv314193770=63, srv1114200426=8, srv580915449=78, srv997803663=99, srv946573301=96, srv2057520330=51, srv1298036119=18, srv1828715867=41, srv1746176898=37, srv1523662010=27, srv894568010=92, srv1874001807=42, srv612537108=81, srv1001647201=0, srv116866562=12, srv1325441163=20, srv381697402=64, srv525107640=74, srv2144146546=57, srv1009767501=1, srv72848611=86, srv1230874047=14, srv2014865861=49, srv109466082=6, srv1900227448=43, srv574423986=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:22,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:22,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:22,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:22,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:22,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:22,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-27 16:53:22,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-27 16:53:22,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-27 16:53:22,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-27 16:53:22,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-27 16:53:22,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-27 16:53:22,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-27 16:53:22,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-27 16:53:22,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-27 16:53:22,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-27 16:53:22,315 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0389095664123722, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05650303171151609, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0389095664123722, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:22,315 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-27 16:53:22,315 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17682839952363114, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0389095664123722, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05650303171151609, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0389095664123722, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-27 16:53:22,602 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations. Found a solution that moves 2651 regions; Going from a computed imbalance of 0.17682839952363114 to a new imbalance of 0.016492806423771683. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020603804810396566); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032773069341672505, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29455555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020603804810396566); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:22,612 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1432689957:80 , srv2095528547:80 , srv48981370:80 , srv702320782:80 , srv749714530:80 , srv946573301:80 , srv1001647201:81 , srv2144146546:81 , srv1351882054:82 , srv2080280625:82 , srv594974274:82 , srv109466082:83 , srv1298036119:83 , srv2072788877:83 , srv247567755:83 , srv261284709:83 , srv1114200426:84 , srv116866562:84 , srv1397352207:84 , srv1573126311:84 , srv1648373453:84 , srv1649642589:84 , srv1709088926:84 , srv1726782423:84 , srv1828715867:84 , srv1909135405:84 , srv1990843597:84 , srv580915449:84 , srv612537108:84 , srv84439471:84 , srv1076899667:85 , srv1124415620:85 , srv1205422369:85 , srv1267412393:85 , srv1315261174:85 , srv1325441163:85 , srv1468239694:85 , srv1506518013:85 , srv1669708836:85 , srv1874001807:85 , srv1992252086:85 , srv2004586079:85 , srv27400749:85 , srv292505542:85 , srv525167318:85 , srv696702450:85 , srv827309:85 , srv1009767501:86 , srv1230874047:86 , srv1270609625:86 , srv1560411052:86 , srv1624909744:86 , srv1900227448:86 , srv2057520330:86 , srv2068256546:86 , srv384570389:86 , srv424806870:86 , srv447178306:86 , srv454432571:86 , srv574423986:86 , srv636444712:86 , srv779786899:86 , srv894568010:86 , srv995373230:86 , srv995563620:86 , srv997803663:86 , srv1092723356:87 , srv1097727376:87 , srv1523662010:87 , srv1746176898:87 , srv2014865861:87 , srv410088693:87 , srv454393975:87 , srv72848611:87 , srv838479219:87 , srv900410742:87 , srv906449752:87 , srv1359459860:88 , srv1812756772:88 , srv408177529:88 , srv459531023:88 , srv525107640:88 , srv578984282:88 , srv1040104660:89 , srv1790788064:89 , srv1808687888:89 , srv311745780:89 , srv381697402:89 , srv721402728:89 , srv1016985188:90 , srv1263841632:90 , srv1601093463:90 , srv1962039392:90 , srv2034518343:90 , srv212235009:90 , srv314193770:90 , srv590960780:90 , srv897095270:90 , srv1123811656:92 , srv1115327261:521 } 2023-12-27 16:53:22,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:22,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2034518343=50, srv1909135405=44, srv2080280625=54, srv212235009=56, srv594974274=80, srv696702450=83, srv1709088926=35, srv578984282=77, srv48981370=73, srv1397352207=23, srv1076899667=4, srv1092723356=5, srv827309=89, srv1115327261=9, srv27400749=60, srv1648373453=32, srv1573126311=29, srv1992252086=47, srv1351882054=21, srv995373230=97, srv1506518013=26, srv525167318=75, srv1790788064=38, srv1468239694=25, srv2068256546=52, srv1263841632=15, srv1267412393=16, srv900410742=94, srv1560411052=28, srv261284709=59, srv838479219=90, srv1601093463=30, srv2095528547=55, srv408177529=66, srv995563620=98, srv384570389=65, srv897095270=93, srv1990843597=46, srv721402728=85, srv906449752=95, srv410088693=67, srv459531023=72, srv1124415620=11, srv247567755=58, srv1123811656=10, srv1624909744=31, srv1270609625=17, srv1432689957=24, srv84439471=91, srv424806870=68, srv702320782=84, srv749714530=87, srv1315261174=19, srv292505542=61, srv2004586079=48, srv1016985188=2, srv1649642589=33, srv1205422369=13, srv590960780=79, srv779786899=88, srv1808687888=39, srv1097727376=7, srv447178306=69, srv1669708836=34, srv2072788877=53, srv1812756772=40, srv1359459860=22, srv1962039392=45, srv1040104660=3, srv636444712=82, srv454393975=70, srv311745780=62, srv454432571=71, srv1726782423=36, srv314193770=63, srv1114200426=8, srv580915449=78, srv997803663=99, srv946573301=96, srv2057520330=51, srv1298036119=18, srv1828715867=41, srv1746176898=37, srv1523662010=27, srv894568010=92, srv1874001807=42, srv612537108=81, srv1001647201=0, srv116866562=12, srv1325441163=20, srv381697402=64, srv525107640=74, srv2144146546=57, srv1009767501=1, srv72848611=86, srv1230874047=14, srv2014865861=49, srv109466082=6, srv1900227448=43, srv574423986=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:22,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:22,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:22,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-27 16:53:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-27 16:53:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-27 16:53:22,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-27 16:53:22,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-27 16:53:22,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-27 16:53:22,797 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020603804810396566); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032773069341672505, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020603804810396566); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:22,797 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-27 16:53:22,797 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016474243725023166, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020603804810396566); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032773069341672505, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020603804810396566); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-27 16:53:23,178 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 522 ms to try 5000 different iterations. Found a solution that moves 1052 regions; Going from a computed imbalance of 0.016474243725023166 to a new imbalance of 5.887348737827802E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00498657631892483); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054843927113124155); MoveCostFunction : (multiplier=7.0, imbalance=0.11688888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00498657631892483); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:23,180 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1123811656:82 , srv2034518343:82 , srv454432571:82 , srv636444712:82 , srv1009767501:83 , srv1114200426:83 , srv1263841632:83 , srv1359459860:83 , srv314193770:83 , srv447178306:83 , srv590960780:83 , srv897095270:83 , srv906449752:83 , srv1016985188:84 , srv1124415620:84 , srv1270609625:84 , srv1560411052:84 , srv1812756772:84 , srv2014865861:84 , srv459531023:84 , srv48981370:84 , srv612537108:84 , srv721402728:84 , srv1076899667:85 , srv1097727376:85 , srv1230874047:85 , srv1468239694:85 , srv1506518013:85 , srv1573126311:85 , srv1601093463:85 , srv1709088926:85 , srv1790788064:85 , srv1900227448:85 , srv1962039392:85 , srv1992252086:85 , srv2004586079:85 , srv2057520330:85 , srv212235009:85 , srv27400749:85 , srv311745780:85 , srv381697402:85 , srv384570389:85 , srv408177529:85 , srv410088693:85 , srv578984282:85 , srv696702450:85 , srv72848611:85 , srv779786899:85 , srv827309:85 , srv838479219:85 , srv894568010:85 , srv946573301:85 , srv995373230:85 , srv1040104660:86 , srv116866562:86 , srv1205422369:86 , srv1624909744:86 , srv1648373453:86 , srv1649642589:86 , srv1669708836:86 , srv1726782423:86 , srv1909135405:86 , srv1990843597:86 , srv2068256546:86 , srv2072788877:86 , srv2080280625:86 , srv2144146546:86 , srv261284709:86 , srv454393975:86 , srv525107640:86 , srv525167318:86 , srv84439471:86 , srv900410742:86 , srv995563620:86 , srv997803663:86 , srv1001647201:87 , srv1092723356:87 , srv1298036119:87 , srv1315261174:87 , srv1397352207:87 , srv1432689957:87 , srv1746176898:87 , srv1808687888:87 , srv1828715867:87 , srv1874001807:87 , srv292505542:87 , srv424806870:87 , srv574423986:87 , srv702320782:87 , srv749714530:87 , srv109466082:88 , srv1325441163:88 , srv1351882054:88 , srv1523662010:88 , srv2095528547:88 , srv247567755:88 , srv580915449:88 , srv594974274:88 , srv1267412393:89 , srv1115327261:545 } 2023-12-27 16:53:23,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:23,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2034518343=50, srv1909135405=44, srv2080280625=54, srv212235009=56, srv594974274=80, srv696702450=83, srv1709088926=35, srv578984282=77, srv48981370=73, srv1397352207=23, srv1076899667=4, srv1092723356=5, srv827309=89, srv1115327261=9, srv27400749=60, srv1648373453=32, srv1573126311=29, srv1992252086=47, srv1351882054=21, srv995373230=97, srv1506518013=26, srv525167318=75, srv1790788064=38, srv1468239694=25, srv2068256546=52, srv1263841632=15, srv1267412393=16, srv900410742=94, srv1560411052=28, srv261284709=59, srv838479219=90, srv1601093463=30, srv2095528547=55, srv408177529=66, srv995563620=98, srv384570389=65, srv897095270=93, srv1990843597=46, srv721402728=85, srv906449752=95, srv410088693=67, srv459531023=72, srv1124415620=11, srv247567755=58, srv1123811656=10, srv1624909744=31, srv1270609625=17, srv1432689957=24, srv84439471=91, srv424806870=68, srv702320782=84, srv749714530=87, srv1315261174=19, srv292505542=61, srv2004586079=48, srv1016985188=2, srv1649642589=33, srv1205422369=13, srv590960780=79, srv779786899=88, srv1808687888=39, srv1097727376=7, srv447178306=69, srv1669708836=34, srv2072788877=53, srv1812756772=40, srv1359459860=22, srv1962039392=45, srv1040104660=3, srv636444712=82, srv454393975=70, srv311745780=62, srv454432571=71, srv1726782423=36, srv314193770=63, srv1114200426=8, srv580915449=78, srv997803663=99, srv946573301=96, srv2057520330=51, srv1298036119=18, srv1828715867=41, srv1746176898=37, srv1523662010=27, srv894568010=92, srv1874001807=42, srv612537108=81, srv1001647201=0, srv116866562=12, srv1325441163=20, srv381697402=64, srv525107640=74, srv2144146546=57, srv1009767501=1, srv72848611=86, srv1230874047=14, srv2014865861=49, srv109466082=6, srv1900227448=43, srv574423986=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:23,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:23,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:23,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:23,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:23,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-27 16:53:23,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-27 16:53:23,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-27 16:53:23,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-27 16:53:23,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-27 16:53:23,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-27 16:53:23,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-27 16:53:23,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-27 16:53:23,348 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00498657631892483); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054843927113124155); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00498657631892483); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:23,348 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-27 16:53:23,349 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.813686123405175E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00498657631892483); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054843927113124155); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00498657631892483); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-27 16:53:23,624 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 398 ms to try 5000 different iterations. Found a solution that moves 203 regions; Going from a computed imbalance of 5.813686123405175E-4 to a new imbalance of 3.3480970724596494E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.022555555555555554); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:23,625 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1267412393:82 , srv1746176898:83 , srv1808687888:83 , srv1962039392:83 , srv408177529:83 , srv410088693:83 , srv1009767501:84 , srv1040104660:84 , srv1092723356:84 , srv1124415620:84 , srv1205422369:84 , srv1230874047:84 , srv1315261174:84 , srv1325441163:84 , srv1351882054:84 , srv1359459860:84 , srv1397352207:84 , srv1432689957:84 , srv1506518013:84 , srv1523662010:84 , srv1573126311:84 , srv1649642589:84 , srv1669708836:84 , srv1709088926:84 , srv1790788064:84 , srv1909135405:84 , srv1992252086:84 , srv2034518343:84 , srv2057520330:84 , srv2068256546:84 , srv2080280625:84 , srv2095528547:84 , srv212235009:84 , srv2144146546:84 , srv247567755:84 , srv261284709:84 , srv311745780:84 , srv314193770:84 , srv381697402:84 , srv424806870:84 , srv447178306:84 , srv454393975:84 , srv48981370:84 , srv525107640:84 , srv525167318:84 , srv594974274:84 , srv721402728:84 , srv72848611:84 , srv749714530:84 , srv946573301:84 , srv995373230:84 , srv995563620:84 , srv1001647201:85 , srv1016985188:85 , srv1076899667:85 , srv109466082:85 , srv1097727376:85 , srv1123811656:85 , srv116866562:85 , srv1263841632:85 , srv1270609625:85 , srv1298036119:85 , srv1468239694:85 , srv1560411052:85 , srv1601093463:85 , srv1624909744:85 , srv1648373453:85 , srv1726782423:85 , srv1812756772:85 , srv1828715867:85 , srv1874001807:85 , srv1900227448:85 , srv1990843597:85 , srv2004586079:85 , srv2014865861:85 , srv2072788877:85 , srv27400749:85 , srv292505542:85 , srv384570389:85 , srv454432571:85 , srv459531023:85 , srv574423986:85 , srv578984282:85 , srv580915449:85 , srv590960780:85 , srv612537108:85 , srv636444712:85 , srv779786899:85 , srv827309:85 , srv838479219:85 , srv84439471:85 , srv894568010:85 , srv900410742:85 , srv906449752:85 , srv997803663:85 , srv1114200426:86 , srv696702450:86 , srv702320782:86 , srv897095270:86 , srv1115327261:640 } 2023-12-27 16:53:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2034518343=50, srv1909135405=44, srv2080280625=54, srv212235009=56, srv594974274=80, srv696702450=83, srv1709088926=35, srv578984282=77, srv48981370=73, srv1397352207=23, srv1076899667=4, srv1092723356=5, srv827309=89, srv1115327261=9, srv27400749=60, srv1648373453=32, srv1573126311=29, srv1992252086=47, srv1351882054=21, srv995373230=97, srv1506518013=26, srv525167318=75, srv1790788064=38, srv1468239694=25, srv2068256546=52, srv1263841632=15, srv1267412393=16, srv900410742=94, srv1560411052=28, srv261284709=59, srv838479219=90, srv1601093463=30, srv2095528547=55, srv408177529=66, srv995563620=98, srv384570389=65, srv897095270=93, srv1990843597=46, srv721402728=85, srv906449752=95, srv410088693=67, srv459531023=72, srv1124415620=11, srv247567755=58, srv1123811656=10, srv1624909744=31, srv1270609625=17, srv1432689957=24, srv84439471=91, srv424806870=68, srv702320782=84, srv749714530=87, srv1315261174=19, srv292505542=61, srv2004586079=48, srv1016985188=2, srv1649642589=33, srv1205422369=13, srv590960780=79, srv779786899=88, srv1808687888=39, srv1097727376=7, srv447178306=69, srv1669708836=34, srv2072788877=53, srv1812756772=40, srv1359459860=22, srv1962039392=45, srv1040104660=3, srv636444712=82, srv454393975=70, srv311745780=62, srv454432571=71, srv1726782423=36, srv314193770=63, srv1114200426=8, srv580915449=78, srv997803663=99, srv946573301=96, srv2057520330=51, srv1298036119=18, srv1828715867=41, srv1746176898=37, srv1523662010=27, srv894568010=92, srv1874001807=42, srv612537108=81, srv1001647201=0, srv116866562=12, srv1325441163=20, srv381697402=64, srv525107640=74, srv2144146546=57, srv1009767501=1, srv72848611=86, srv1230874047=14, srv2014865861=49, srv109466082=6, srv1900227448=43, srv574423986=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:23,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:23,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:23,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:23,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:23,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:23,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-27 16:53:23,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-27 16:53:23,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-27 16:53:23,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-27 16:53:23,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-27 16:53:23,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-27 16:53:23,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-27 16:53:23,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-27 16:53:23,755 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:23,756 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-27 16:53:23,756 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.205953434362753E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-27 16:53:24,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 354 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 3.205953434362753E-5 to a new imbalance of 5.405989317391374E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:24,009 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1016985188:83 , srv1624909744:83 , srv580915449:83 , srv696702450:83 , srv702320782:83 , srv897095270:83 , srv1001647201:84 , srv1040104660:84 , srv1076899667:84 , srv1092723356:84 , srv109466082:84 , srv1097727376:84 , srv1114200426:84 , srv1123811656:84 , srv1124415620:84 , srv116866562:84 , srv1205422369:84 , srv1230874047:84 , srv1263841632:84 , srv1267412393:84 , srv1270609625:84 , srv1298036119:84 , srv1315261174:84 , srv1325441163:84 , srv1351882054:84 , srv1359459860:84 , srv1397352207:84 , srv1432689957:84 , srv1468239694:84 , srv1506518013:84 , srv1523662010:84 , srv1560411052:84 , srv1601093463:84 , srv1648373453:84 , srv1649642589:84 , srv1669708836:84 , srv1709088926:84 , srv1726782423:84 , srv1790788064:84 , srv1808687888:84 , srv1812756772:84 , srv1828715867:84 , srv1874001807:84 , srv1900227448:84 , srv1909135405:84 , srv1990843597:84 , srv1992252086:84 , srv2004586079:84 , srv2014865861:84 , srv2034518343:84 , srv2057520330:84 , srv2072788877:84 , srv2080280625:84 , srv2095528547:84 , srv212235009:84 , srv247567755:84 , srv261284709:84 , srv27400749:84 , srv292505542:84 , srv311745780:84 , srv314193770:84 , srv381697402:84 , srv384570389:84 , srv408177529:84 , srv410088693:84 , srv424806870:84 , srv447178306:84 , srv454393975:84 , srv454432571:84 , srv459531023:84 , srv48981370:84 , srv525107640:84 , srv525167318:84 , srv574423986:84 , srv578984282:84 , srv590960780:84 , srv594974274:84 , srv612537108:84 , srv636444712:84 , srv721402728:84 , srv72848611:84 , srv749714530:84 , srv779786899:84 , srv827309:84 , srv838479219:84 , srv84439471:84 , srv894568010:84 , srv900410742:84 , srv906449752:84 , srv946573301:84 , srv995373230:84 , srv995563620:84 , srv997803663:84 , srv1009767501:85 , srv1573126311:85 , srv1746176898:85 , srv1962039392:85 , srv2068256546:85 , srv2144146546:85 , srv1115327261:684 } 2023-12-27 16:53:24,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:24,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2034518343=50, srv1909135405=44, srv2080280625=54, srv212235009=56, srv594974274=80, srv696702450=83, srv1709088926=35, srv578984282=77, srv48981370=73, srv1397352207=23, srv1076899667=4, srv1092723356=5, srv827309=89, srv1115327261=9, srv27400749=60, srv1648373453=32, srv1573126311=29, srv1992252086=47, srv1351882054=21, srv995373230=97, srv1506518013=26, srv525167318=75, srv1790788064=38, srv1468239694=25, srv2068256546=52, srv1263841632=15, srv1267412393=16, srv900410742=94, srv1560411052=28, srv261284709=59, srv838479219=90, srv1601093463=30, srv2095528547=55, srv408177529=66, srv995563620=98, srv384570389=65, srv897095270=93, srv1990843597=46, srv721402728=85, srv906449752=95, srv410088693=67, srv459531023=72, srv1124415620=11, srv247567755=58, srv1123811656=10, srv1624909744=31, srv1270609625=17, srv1432689957=24, srv84439471=91, srv424806870=68, srv702320782=84, srv749714530=87, srv1315261174=19, srv292505542=61, srv2004586079=48, srv1016985188=2, srv1649642589=33, srv1205422369=13, srv590960780=79, srv779786899=88, srv1808687888=39, srv1097727376=7, srv447178306=69, srv1669708836=34, srv2072788877=53, srv1812756772=40, srv1359459860=22, srv1962039392=45, srv1040104660=3, srv636444712=82, srv454393975=70, srv311745780=62, srv454432571=71, srv1726782423=36, srv314193770=63, srv1114200426=8, srv580915449=78, srv997803663=99, srv946573301=96, srv2057520330=51, srv1298036119=18, srv1828715867=41, srv1746176898=37, srv1523662010=27, srv894568010=92, srv1874001807=42, srv612537108=81, srv1001647201=0, srv116866562=12, srv1325441163=20, srv381697402=64, srv525107640=74, srv2144146546=57, srv1009767501=1, srv72848611=86, srv1230874047=14, srv2014865861=49, srv109466082=6, srv1900227448=43, srv574423986=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:24,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:24,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:24,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:24,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-27 16:53:24,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-27 16:53:24,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-27 16:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-27 16:53:24,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-27 16:53:24,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.223933426232788E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:24,136 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1016985188:83 , srv1624909744:83 , srv580915449:83 , srv696702450:83 , srv702320782:83 , srv897095270:83 , srv1001647201:84 , srv1040104660:84 , srv1076899667:84 , srv1092723356:84 , srv109466082:84 , srv1097727376:84 , srv1114200426:84 , srv1123811656:84 , srv1124415620:84 , srv116866562:84 , srv1205422369:84 , srv1230874047:84 , srv1263841632:84 , srv1267412393:84 , srv1270609625:84 , srv1298036119:84 , srv1315261174:84 , srv1325441163:84 , srv1351882054:84 , srv1359459860:84 , srv1397352207:84 , srv1432689957:84 , srv1468239694:84 , srv1506518013:84 , srv1523662010:84 , srv1560411052:84 , srv1601093463:84 , srv1648373453:84 , srv1649642589:84 , srv1669708836:84 , srv1709088926:84 , srv1726782423:84 , srv1790788064:84 , srv1808687888:84 , srv1812756772:84 , srv1828715867:84 , srv1874001807:84 , srv1900227448:84 , srv1909135405:84 , srv1990843597:84 , srv1992252086:84 , srv2004586079:84 , srv2014865861:84 , srv2034518343:84 , srv2057520330:84 , srv2072788877:84 , srv2080280625:84 , srv2095528547:84 , srv212235009:84 , srv247567755:84 , srv261284709:84 , srv27400749:84 , srv292505542:84 , srv311745780:84 , srv314193770:84 , srv381697402:84 , srv384570389:84 , srv408177529:84 , srv410088693:84 , srv424806870:84 , srv447178306:84 , srv454393975:84 , srv454432571:84 , srv459531023:84 , srv48981370:84 , srv525107640:84 , srv525167318:84 , srv574423986:84 , srv578984282:84 , srv590960780:84 , srv594974274:84 , srv612537108:84 , srv636444712:84 , srv721402728:84 , srv72848611:84 , srv749714530:84 , srv779786899:84 , srv827309:84 , srv838479219:84 , srv84439471:84 , srv894568010:84 , srv900410742:84 , srv906449752:84 , srv946573301:84 , srv995373230:84 , srv995563620:84 , srv997803663:84 , srv1009767501:85 , srv1573126311:85 , srv1746176898:85 , srv1962039392:85 , srv2068256546:85 , srv2144146546:85 , srv1115327261:684 } 2023-12-27 16:53:24,169 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=294 (was 302), ProcessCount=172 (was 172), AvailableMemoryMB=10901 (was 11538) 2023-12-27 16:53:24,178 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=294, ProcessCount=172, AvailableMemoryMB=10898 2023-12-27 16:53:24,179 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-27 16:53:24,179 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-27 16:53:24,196 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100315967:84 , srv1057476199:84 , srv1324259549:84 , srv1383062557:84 , srv153557834:84 , srv1631200888:84 , srv1632393967:84 , srv165801461:84 , srv1768972374:84 , srv1795356620:84 , srv1822386772:84 , srv1873029564:84 , srv1875578115:84 , srv1885959460:84 , srv2088595656:84 , srv2112413246:84 , srv2130884657:84 , srv237072232:84 , srv326493701:84 , srv344789480:84 , srv483566065:84 , srv495065685:84 , srv596534803:84 , srv611125748:84 , srv626278539:84 , srv657670373:84 , srv699167472:84 , srv714191310:84 , srv845024613:84 , srv347074101:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-27 16:53:24,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:53:24,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2088595656=14, srv714191310=28, srv1822386772=10, srv1885959460=13, srv100315967=0, srv153557834=4, srv237072232=17, srv1383062557=3, srv596534803=23, srv1324259549=2, srv611125748=24, srv699167472=27, srv1631200888=5, srv2112413246=15, srv1057476199=1, srv2130884657=16, srv165801461=7, srv1795356620=9, srv626278539=25, srv657670373=26, srv845024613=29, srv326493701=18, srv495065685=22, srv1768972374=8, srv347074101=20, srv483566065=21, srv344789480=19, srv1632393967=6, srv1873029564=11, srv1875578115=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:53:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:53:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:53:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-27 16:53:24,229 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1235743959850746, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:53:24,229 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912582422150954, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1235743959850746, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-27 16:55:24,215 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5353182 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912582422150954 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:55:24,221 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv100315967:90 , srv1057476199:90 , srv1324259549:90 , srv1383062557:90 , srv153557834:90 , srv1631200888:90 , srv1632393967:90 , srv165801461:90 , srv1768972374:90 , srv1795356620:90 , srv1822386772:90 , srv1873029564:90 , srv1875578115:90 , srv1885959460:90 , srv2088595656:90 , srv2112413246:90 , srv2130884657:90 , srv237072232:90 , srv326493701:90 , srv344789480:90 , srv347074101:90 , srv483566065:90 , srv495065685:90 , srv596534803:90 , srv611125748:90 , srv626278539:90 , srv657670373:90 , srv699167472:90 , srv714191310:90 , srv845024613:90 } 2023-12-27 16:55:24,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-27 16:55:24,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2088595656=14, srv714191310=28, srv1822386772=10, srv1885959460=13, srv100315967=0, srv153557834=4, srv237072232=17, srv1383062557=3, srv596534803=23, srv1324259549=2, srv611125748=24, srv699167472=27, srv1631200888=5, srv2112413246=15, srv1057476199=1, srv2130884657=16, srv165801461=7, srv1795356620=9, srv626278539=25, srv657670373=26, srv845024613=29, srv326493701=18, srv495065685=22, srv1768972374=8, srv347074101=20, srv483566065=21, srv344789480=19, srv1632393967=6, srv1873029564=11, srv1875578115=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-27 16:55:24,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-27 16:55:24,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-27 16:55:24,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-27 16:55:24,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-27 16:55:24,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-27 16:55:24,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-27 16:55:24,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-27 16:55:24,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-27 16:55:24,238 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-27 16:55:24,238 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv100315967:90 , srv1057476199:90 , srv1324259549:90 , srv1383062557:90 , srv153557834:90 , srv1631200888:90 , srv1632393967:90 , srv165801461:90 , srv1768972374:90 , srv1795356620:90 , srv1822386772:90 , srv1873029564:90 , srv1875578115:90 , srv1885959460:90 , srv2088595656:90 , srv2112413246:90 , srv2130884657:90 , srv237072232:90 , srv326493701:90 , srv344789480:90 , srv347074101:90 , srv483566065:90 , srv495065685:90 , srv596534803:90 , srv611125748:90 , srv626278539:90 , srv657670373:90 , srv699167472:90 , srv714191310:90 , srv845024613:90 } 2023-12-27 16:55:24,250 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=193 (was 294), ProcessCount=170 (was 172), AvailableMemoryMB=10569 (was 10898)