2023-07-18 05:53:11,489 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-18 05:53:11,808 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-18 05:53:11,919 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-18 05:53:11,985 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 05:53:11,993 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 05:53:12,008 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=257, ProcessCount=174, AvailableMemoryMB=7524 2023-07-18 05:53:12,010 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 05:53:12,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 05:53:12,161 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005333102:84 , srv1008978453:84 , srv10101954:84 , srv1084229289:84 , srv1102901495:84 , srv1138111820:84 , srv1150206193:84 , srv1175109797:84 , srv1194223750:84 , srv1213040339:84 , srv1225231914:84 , srv1233718715:84 , srv1248880618:84 , srv1273466838:84 , srv1284718475:84 , srv1300369261:84 , srv1322221120:84 , srv1340773604:84 , srv1350107038:84 , srv1359291857:84 , srv1379203835:84 , srv1399927783:84 , srv1442054582:84 , srv1470818325:84 , srv1474027827:84 , srv1494692360:84 , srv1503513242:84 , srv1522958748:84 , srv152885460:84 , srv1538207779:84 , srv1542769958:84 , srv1543687741:84 , srv1608357750:84 , srv1644682694:84 , srv1708265297:84 , srv1729870646:84 , srv1810958243:84 , srv1818438161:84 , srv1821581544:84 , srv1841847373:84 , srv1855907881:84 , srv1864565860:84 , srv1866234766:84 , srv1911192861:84 , srv1913791995:84 , srv1920347198:84 , srv1944493647:84 , srv1957063474:84 , srv2045559203:84 , srv2049343245:84 , srv2075945332:84 , srv2084016653:84 , srv208725600:84 , srv2098843663:84 , srv221960764:84 , srv223856161:84 , srv267658464:84 , srv277506339:84 , srv312457303:84 , srv442262948:84 , srv450387239:84 , srv452572441:84 , srv473896525:84 , srv474635431:84 , srv475090900:84 , srv48097626:84 , srv488763520:84 , srv512649340:84 , srv531474607:84 , srv532846838:84 , srv535839589:84 , srv551360278:84 , srv554471856:84 , srv58126619:84 , srv581803984:84 , srv58335656:84 , srv595814832:84 , srv608568418:84 , srv615374544:84 , srv617419927:84 , srv620694243:84 , srv6275769:84 , srv628358055:84 , srv645769324:84 , srv662222376:84 , srv66435335:84 , srv666726558:84 , srv674949784:84 , srv707343117:84 , srv722283793:84 , srv728373151:84 , srv818328858:84 , srv869431368:84 , srv884720679:84 , srv896189683:84 , srv898457340:84 , srv902522733:84 , srv928669290:84 , srv99140737:84 , srv755815885:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-18 05:53:12,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:53:12,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896189683=95, srv1213040339=9, srv707343117=88, srv902522733=97, srv1084229289=3, srv442262948=59, srv1322221120=16, srv2084016653=51, srv277506339=57, srv2098843663=53, srv474635431=63, srv1248880618=12, srv1944493647=46, srv617419927=79, srv535839589=70, srv1543687741=31, srv1821581544=38, srv2075945332=50, srv1708265297=34, srv1503513242=26, srv1818438161=37, srv1957063474=47, srv674949784=87, srv1300369261=15, srv1379203835=20, srv1542769958=30, srv818328858=92, srv595814832=76, srv1359291857=19, srv1644682694=33, srv452572441=61, srv1470818325=23, srv1008978453=1, srv1102901495=4, srv58335656=75, srv48097626=65, srv475090900=64, srv645769324=83, srv1474027827=24, srv1150206193=6, srv1350107038=18, srv608568418=77, srv554471856=72, srv666726558=86, srv551360278=71, srv1841847373=39, srv1284718475=14, srv488763520=66, srv662222376=84, srv728373151=90, srv1864565860=41, srv581803984=74, srv1399927783=21, srv531474607=68, srv152885460=28, srv221960764=54, srv722283793=89, srv1273466838=13, srv223856161=55, srv512649340=67, srv58126619=73, srv628358055=82, srv99140737=99, srv1911192861=43, srv66435335=85, srv1855907881=40, srv1913791995=44, srv1810958243=36, srv10101954=2, srv1920347198=45, srv1340773604=17, srv1494692360=25, srv1522958748=27, srv312457303=58, srv2049343245=49, srv615374544=78, srv450387239=60, srv1608357750=32, srv1225231914=10, srv6275769=81, srv1005333102=0, srv884720679=94, srv755815885=91, srv1866234766=42, srv473896525=62, srv928669290=98, srv1233718715=11, srv869431368=93, srv2045559203=48, srv1175109797=7, srv532846838=69, srv1729870646=35, srv1538207779=29, srv1138111820=5, srv267658464=56, srv620694243=80, srv1194223750=8, srv898457340=96, srv1442054582=22, srv208725600=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:53:12,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:53:12,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:53:12,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:53:12,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 05:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 05:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 05:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 05:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 05:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 05:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 05:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 05:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 05:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 05:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 05:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 05:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 05:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 05:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 05:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 05:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 05:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 05:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 05:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 05:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 05:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 05:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 05:53:12,436 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:12,436 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 05:53:12,437 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 05:53:12,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 471 ms to try 5000 different iterations. Found a solution that moves 4751 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17580702162914266. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0381218241365192, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05471895339670275, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5278888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0381218241365192, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2385, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:12,791 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1944493647:81 , srv1084229289:82 , srv1359291857:82 , srv267658464:82 , srv99140737:82 , srv1150206193:83 , srv152885460:83 , srv1913791995:83 , srv2049343245:83 , srv551360278:83 , srv595814832:83 , srv1818438161:84 , srv1821581544:84 , srv1864565860:84 , srv535839589:84 , srv1233718715:85 , srv1340773604:85 , srv1379203835:85 , srv2075945332:85 , srv531474607:85 , srv58335656:85 , srv6275769:85 , srv66435335:85 , srv884720679:85 , srv1008978453:86 , srv10101954:86 , srv1138111820:86 , srv1273466838:86 , srv1284718475:86 , srv1350107038:86 , srv1399927783:86 , srv1708265297:86 , srv1810958243:86 , srv1855907881:86 , srv1866234766:86 , srv2045559203:86 , srv221960764:86 , srv442262948:86 , srv452572441:86 , srv473896525:86 , srv475090900:86 , srv488763520:86 , srv532846838:86 , srv707343117:86 , srv728373151:86 , srv898457340:86 , srv1194223750:87 , srv1300369261:87 , srv1474027827:87 , srv1503513242:87 , srv1522958748:87 , srv1608357750:87 , srv1729870646:87 , srv1841847373:87 , srv2098843663:87 , srv312457303:87 , srv450387239:87 , srv48097626:87 , srv554471856:87 , srv581803984:87 , srv608568418:87 , srv615374544:87 , srv620694243:87 , srv628358055:87 , srv645769324:87 , srv674949784:87 , srv869431368:87 , srv902522733:87 , srv1213040339:88 , srv1225231914:88 , srv1248880618:88 , srv1442054582:88 , srv1470818325:88 , srv1543687741:88 , srv1911192861:88 , srv1920347198:88 , srv208725600:88 , srv223856161:88 , srv277506339:88 , srv818328858:88 , srv928669290:88 , srv1005333102:89 , srv1102901495:89 , srv1175109797:89 , srv1494692360:89 , srv1542769958:89 , srv474635431:89 , srv58126619:89 , srv617419927:89 , srv666726558:89 , srv722283793:89 , srv1322221120:90 , srv1538207779:90 , srv1644682694:90 , srv2084016653:90 , srv662222376:90 , srv896189683:90 , srv512649340:91 , srv1957063474:92 , srv755815885:429 } 2023-07-18 05:53:12,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:53:12,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896189683=95, srv1213040339=9, srv707343117=88, srv902522733=97, srv1084229289=3, srv442262948=59, srv1322221120=16, srv2084016653=51, srv277506339=57, srv2098843663=53, srv474635431=63, srv1248880618=12, srv1944493647=46, srv617419927=79, srv535839589=70, srv1543687741=31, srv1821581544=38, srv2075945332=50, srv1708265297=34, srv1503513242=26, srv1818438161=37, srv1957063474=47, srv674949784=87, srv1300369261=15, srv1379203835=20, srv1542769958=30, srv818328858=92, srv595814832=76, srv1359291857=19, srv1644682694=33, srv452572441=61, srv1470818325=23, srv1008978453=1, srv1102901495=4, srv58335656=75, srv48097626=65, srv475090900=64, srv645769324=83, srv1474027827=24, srv1150206193=6, srv1350107038=18, srv608568418=77, srv554471856=72, srv666726558=86, srv551360278=71, srv1841847373=39, srv1284718475=14, srv488763520=66, srv662222376=84, srv728373151=90, srv1864565860=41, srv581803984=74, srv1399927783=21, srv531474607=68, srv152885460=28, srv221960764=54, srv722283793=89, srv1273466838=13, srv223856161=55, srv512649340=67, srv58126619=73, srv628358055=82, srv99140737=99, srv1911192861=43, srv66435335=85, srv1855907881=40, srv1913791995=44, srv1810958243=36, srv10101954=2, srv1920347198=45, srv1340773604=17, srv1494692360=25, srv1522958748=27, srv312457303=58, srv2049343245=49, srv615374544=78, srv450387239=60, srv1608357750=32, srv1225231914=10, srv6275769=81, srv1005333102=0, srv884720679=94, srv755815885=91, srv1866234766=42, srv473896525=62, srv928669290=98, srv1233718715=11, srv869431368=93, srv2045559203=48, srv1175109797=7, srv532846838=69, srv1729870646=35, srv1538207779=29, srv1138111820=5, srv267658464=56, srv620694243=80, srv1194223750=8, srv898457340=96, srv1442054582=22, srv208725600=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:53:12,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:53:12,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:53:12,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:53:12,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:53:12,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:53:12,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:53:12,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:53:12,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:53:12,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:53:12,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 05:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 05:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 05:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 05:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 05:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 05:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 05:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 05:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 05:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 05:53:12,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0381218241365192, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05471895339670275, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0381218241365192, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2385, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:12,978 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 05:53:12,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17577375441610824, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0381218241365192, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05471895339670275, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0381218241365192, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2385, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 05:53:13,241 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 386 ms to try 5000 different iterations. Found a solution that moves 2655 regions; Going from a computed imbalance of 0.17577375441610824 to a new imbalance of 0.014372043140113876. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020383528043207227); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02853316340929295, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.295, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020383528043207227); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:13,253 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1322221120:80 , srv1644682694:80 , srv1005333102:81 , srv1102901495:81 , srv1957063474:81 , srv512649340:81 , srv662222376:81 , srv1175109797:82 , srv1213040339:82 , srv1248880618:82 , srv1542769958:82 , srv48097626:82 , srv58126619:82 , srv869431368:82 , srv1442054582:83 , srv1522958748:83 , srv1729870646:83 , srv2084016653:83 , srv554471856:83 , srv620694243:83 , srv628358055:83 , srv666726558:83 , srv674949784:83 , srv818328858:83 , srv1470818325:84 , srv1538207779:84 , srv1543687741:84 , srv1841847373:84 , srv1920347198:84 , srv208725600:84 , srv223856161:84 , srv722283793:84 , srv10101954:85 , srv1225231914:85 , srv1300369261:85 , srv1474027827:85 , srv1494692360:85 , srv1608357750:85 , srv2098843663:85 , srv277506339:85 , srv450387239:85 , srv532846838:85 , srv608568418:85 , srv617419927:85 , srv896189683:85 , srv1284718475:86 , srv1399927783:86 , srv1708265297:86 , srv1866234766:86 , srv1911192861:86 , srv2045559203:86 , srv221960764:86 , srv312457303:86 , srv442262948:86 , srv474635431:86 , srv488763520:86 , srv615374544:86 , srv6275769:86 , srv645769324:86 , srv728373151:86 , srv884720679:86 , srv902522733:86 , srv1138111820:87 , srv1194223750:87 , srv1350107038:87 , srv1379203835:87 , srv1503513242:87 , srv1810958243:87 , srv1821581544:87 , srv1855907881:87 , srv475090900:87 , srv581803984:87 , srv898457340:87 , srv1008978453:88 , srv1150206193:88 , srv1233718715:88 , srv1340773604:88 , srv1864565860:88 , srv2075945332:88 , srv452572441:88 , srv473896525:88 , srv531474607:88 , srv535839589:88 , srv551360278:88 , srv58335656:88 , srv595814832:88 , srv66435335:88 , srv1359291857:89 , srv1818438161:89 , srv1913791995:89 , srv267658464:89 , srv707343117:89 , srv928669290:89 , srv99140737:89 , srv152885460:90 , srv2049343245:90 , srv1273466838:91 , srv1944493647:91 , srv1084229289:92 , srv755815885:526 } 2023-07-18 05:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896189683=95, srv1213040339=9, srv707343117=88, srv902522733=97, srv1084229289=3, srv442262948=59, srv1322221120=16, srv2084016653=51, srv277506339=57, srv2098843663=53, srv474635431=63, srv1248880618=12, srv1944493647=46, srv617419927=79, srv535839589=70, srv1543687741=31, srv1821581544=38, srv2075945332=50, srv1708265297=34, srv1503513242=26, srv1818438161=37, srv1957063474=47, srv674949784=87, srv1300369261=15, srv1379203835=20, srv1542769958=30, srv818328858=92, srv595814832=76, srv1359291857=19, srv1644682694=33, srv452572441=61, srv1470818325=23, srv1008978453=1, srv1102901495=4, srv58335656=75, srv48097626=65, srv475090900=64, srv645769324=83, srv1474027827=24, srv1150206193=6, srv1350107038=18, srv608568418=77, srv554471856=72, srv666726558=86, srv551360278=71, srv1841847373=39, srv1284718475=14, srv488763520=66, srv662222376=84, srv728373151=90, srv1864565860=41, srv581803984=74, srv1399927783=21, srv531474607=68, srv152885460=28, srv221960764=54, srv722283793=89, srv1273466838=13, srv223856161=55, srv512649340=67, srv58126619=73, srv628358055=82, srv99140737=99, srv1911192861=43, srv66435335=85, srv1855907881=40, srv1913791995=44, srv1810958243=36, srv10101954=2, srv1920347198=45, srv1340773604=17, srv1494692360=25, srv1522958748=27, srv312457303=58, srv2049343245=49, srv615374544=78, srv450387239=60, srv1608357750=32, srv1225231914=10, srv6275769=81, srv1005333102=0, srv884720679=94, srv755815885=91, srv1866234766=42, srv473896525=62, srv928669290=98, srv1233718715=11, srv869431368=93, srv2045559203=48, srv1175109797=7, srv532846838=69, srv1729870646=35, srv1538207779=29, srv1138111820=5, srv267658464=56, srv620694243=80, srv1194223750=8, srv898457340=96, srv1442054582=22, srv208725600=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:53:13,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:53:13,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:53:13,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 05:53:13,416 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020383528043207227); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02853316340929295, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020383528043207227); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:13,417 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 05:53:13,417 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014353452432766721, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020383528043207227); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02853316340929295, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020383528043207227); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 05:53:13,679 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 374 ms to try 5000 different iterations. Found a solution that moves 975 regions; Going from a computed imbalance of 0.014353452432766721 to a new imbalance of 3.0507101162915247E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005318160234783104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005565647399578146); MoveCostFunction : (multiplier=7.0, imbalance=0.10833333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005318160234783104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:13,681 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv928669290:80 , srv1273466838:81 , srv896189683:82 , srv1084229289:83 , srv312457303:83 , srv474635431:83 , srv707343117:83 , srv1138111820:84 , srv1194223750:84 , srv1225231914:84 , srv1233718715:84 , srv1350107038:84 , srv1494692360:84 , srv1503513242:84 , srv1538207779:84 , srv1911192861:84 , srv2084016653:84 , srv452572441:84 , srv473896525:84 , srv475090900:84 , srv531474607:84 , srv581803984:84 , srv617419927:84 , srv645769324:84 , srv728373151:84 , srv898457340:84 , srv1008978453:85 , srv1300369261:85 , srv1340773604:85 , srv1399927783:85 , srv1470818325:85 , srv152885460:85 , srv1708265297:85 , srv1810958243:85 , srv1818438161:85 , srv1855907881:85 , srv1866234766:85 , srv1913791995:85 , srv1944493647:85 , srv1957063474:85 , srv2045559203:85 , srv2049343245:85 , srv2075945332:85 , srv221960764:85 , srv223856161:85 , srv277506339:85 , srv442262948:85 , srv488763520:85 , srv512649340:85 , srv551360278:85 , srv58335656:85 , srv608568418:85 , srv615374544:85 , srv66435335:85 , srv666726558:85 , srv722283793:85 , srv902522733:85 , srv10101954:86 , srv1175109797:86 , srv1284718475:86 , srv1359291857:86 , srv1379203835:86 , srv1442054582:86 , srv1474027827:86 , srv1543687741:86 , srv1608357750:86 , srv1821581544:86 , srv1841847373:86 , srv1864565860:86 , srv1920347198:86 , srv208725600:86 , srv2098843663:86 , srv450387239:86 , srv535839589:86 , srv58126619:86 , srv595814832:86 , srv662222376:86 , srv99140737:86 , srv1102901495:87 , srv1150206193:87 , srv1213040339:87 , srv1522958748:87 , srv1644682694:87 , srv1729870646:87 , srv267658464:87 , srv532846838:87 , srv6275769:87 , srv628358055:87 , srv674949784:87 , srv818328858:87 , srv884720679:87 , srv1005333102:88 , srv1248880618:88 , srv1542769958:88 , srv554471856:88 , srv620694243:88 , srv1322221120:89 , srv48097626:89 , srv869431368:89 , srv755815885:550 } 2023-07-18 05:53:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:53:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896189683=95, srv1213040339=9, srv707343117=88, srv902522733=97, srv1084229289=3, srv442262948=59, srv1322221120=16, srv2084016653=51, srv277506339=57, srv2098843663=53, srv474635431=63, srv1248880618=12, srv1944493647=46, srv617419927=79, srv535839589=70, srv1543687741=31, srv1821581544=38, srv2075945332=50, srv1708265297=34, srv1503513242=26, srv1818438161=37, srv1957063474=47, srv674949784=87, srv1300369261=15, srv1379203835=20, srv1542769958=30, srv818328858=92, srv595814832=76, srv1359291857=19, srv1644682694=33, srv452572441=61, srv1470818325=23, srv1008978453=1, srv1102901495=4, srv58335656=75, srv48097626=65, srv475090900=64, srv645769324=83, srv1474027827=24, srv1150206193=6, srv1350107038=18, srv608568418=77, srv554471856=72, srv666726558=86, srv551360278=71, srv1841847373=39, srv1284718475=14, srv488763520=66, srv662222376=84, srv728373151=90, srv1864565860=41, srv581803984=74, srv1399927783=21, srv531474607=68, srv152885460=28, srv221960764=54, srv722283793=89, srv1273466838=13, srv223856161=55, srv512649340=67, srv58126619=73, srv628358055=82, srv99140737=99, srv1911192861=43, srv66435335=85, srv1855907881=40, srv1913791995=44, srv1810958243=36, srv10101954=2, srv1920347198=45, srv1340773604=17, srv1494692360=25, srv1522958748=27, srv312457303=58, srv2049343245=49, srv615374544=78, srv450387239=60, srv1608357750=32, srv1225231914=10, srv6275769=81, srv1005333102=0, srv884720679=94, srv755815885=91, srv1866234766=42, srv473896525=62, srv928669290=98, srv1233718715=11, srv869431368=93, srv2045559203=48, srv1175109797=7, srv532846838=69, srv1729870646=35, srv1538207779=29, srv1138111820=5, srv267658464=56, srv620694243=80, srv1194223750=8, srv898457340=96, srv1442054582=22, srv208725600=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:53:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:53:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:53:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 05:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 05:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 05:53:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 05:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 05:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 05:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 05:53:13,820 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005318160234783104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005565647399578146); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005318160234783104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:13,820 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 05:53:13,821 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.982439157107055E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005318160234783104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005565647399578146); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005318160234783104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 05:53:14,209 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 496 ms to try 5000 different iterations. Found a solution that moves 186 regions; Going from a computed imbalance of 2.982439157107055E-4 to a new imbalance of 1.2501959363512222E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.020666666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:14,210 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1008978453:83 , srv1322221120:83 , srv1542769958:83 , srv1005333102:84 , srv1084229289:84 , srv1150206193:84 , srv1194223750:84 , srv1248880618:84 , srv1273466838:84 , srv1284718475:84 , srv1300369261:84 , srv1340773604:84 , srv1474027827:84 , srv1494692360:84 , srv1503513242:84 , srv1522958748:84 , srv152885460:84 , srv1538207779:84 , srv1543687741:84 , srv1608357750:84 , srv1810958243:84 , srv1818438161:84 , srv1855907881:84 , srv1864565860:84 , srv1911192861:84 , srv1920347198:84 , srv1957063474:84 , srv2049343245:84 , srv2075945332:84 , srv208725600:84 , srv2098843663:84 , srv223856161:84 , srv277506339:84 , srv442262948:84 , srv450387239:84 , srv452572441:84 , srv473896525:84 , srv474635431:84 , srv48097626:84 , srv532846838:84 , srv535839589:84 , srv554471856:84 , srv581803984:84 , srv58335656:84 , srv615374544:84 , srv617419927:84 , srv620694243:84 , srv6275769:84 , srv66435335:84 , srv666726558:84 , srv674949784:84 , srv707343117:84 , srv722283793:84 , srv818328858:84 , srv869431368:84 , srv884720679:84 , srv896189683:84 , srv902522733:84 , srv10101954:85 , srv1138111820:85 , srv1175109797:85 , srv1213040339:85 , srv1233718715:85 , srv1350107038:85 , srv1359291857:85 , srv1379203835:85 , srv1399927783:85 , srv1442054582:85 , srv1470818325:85 , srv1644682694:85 , srv1708265297:85 , srv1729870646:85 , srv1821581544:85 , srv1841847373:85 , srv1866234766:85 , srv1913791995:85 , srv1944493647:85 , srv2045559203:85 , srv2084016653:85 , srv221960764:85 , srv267658464:85 , srv475090900:85 , srv512649340:85 , srv58126619:85 , srv595814832:85 , srv608568418:85 , srv628358055:85 , srv645769324:85 , srv928669290:85 , srv99140737:85 , srv1102901495:86 , srv1225231914:86 , srv312457303:86 , srv488763520:86 , srv531474607:86 , srv551360278:86 , srv662222376:86 , srv728373151:86 , srv898457340:86 , srv755815885:637 } 2023-07-18 05:53:14,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:53:14,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896189683=95, srv1213040339=9, srv707343117=88, srv902522733=97, srv1084229289=3, srv442262948=59, srv1322221120=16, srv2084016653=51, srv277506339=57, srv2098843663=53, srv474635431=63, srv1248880618=12, srv1944493647=46, srv617419927=79, srv535839589=70, srv1543687741=31, srv1821581544=38, srv2075945332=50, srv1708265297=34, srv1503513242=26, srv1818438161=37, srv1957063474=47, srv674949784=87, srv1300369261=15, srv1379203835=20, srv1542769958=30, srv818328858=92, srv595814832=76, srv1359291857=19, srv1644682694=33, srv452572441=61, srv1470818325=23, srv1008978453=1, srv1102901495=4, srv58335656=75, srv48097626=65, srv475090900=64, srv645769324=83, srv1474027827=24, srv1150206193=6, srv1350107038=18, srv608568418=77, srv554471856=72, srv666726558=86, srv551360278=71, srv1841847373=39, srv1284718475=14, srv488763520=66, srv662222376=84, srv728373151=90, srv1864565860=41, srv581803984=74, srv1399927783=21, srv531474607=68, srv152885460=28, srv221960764=54, srv722283793=89, srv1273466838=13, srv223856161=55, srv512649340=67, srv58126619=73, srv628358055=82, srv99140737=99, srv1911192861=43, srv66435335=85, srv1855907881=40, srv1913791995=44, srv1810958243=36, srv10101954=2, srv1920347198=45, srv1340773604=17, srv1494692360=25, srv1522958748=27, srv312457303=58, srv2049343245=49, srv615374544=78, srv450387239=60, srv1608357750=32, srv1225231914=10, srv6275769=81, srv1005333102=0, srv884720679=94, srv755815885=91, srv1866234766=42, srv473896525=62, srv928669290=98, srv1233718715=11, srv869431368=93, srv2045559203=48, srv1175109797=7, srv532846838=69, srv1729870646=35, srv1538207779=29, srv1138111820=5, srv267658464=56, srv620694243=80, srv1194223750=8, srv898457340=96, srv1442054582=22, srv208725600=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 05:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 05:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 05:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 05:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 05:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 05:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 05:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 05:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 05:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 05:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 05:53:14,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 05:53:14,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 05:53:14,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 05:53:14,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 05:53:14,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 05:53:14,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 05:53:14,373 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.119955952676234E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:14,373 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1008978453:83 , srv1322221120:83 , srv1542769958:83 , srv1005333102:84 , srv1084229289:84 , srv1150206193:84 , srv1194223750:84 , srv1248880618:84 , srv1273466838:84 , srv1284718475:84 , srv1300369261:84 , srv1340773604:84 , srv1474027827:84 , srv1494692360:84 , srv1503513242:84 , srv1522958748:84 , srv152885460:84 , srv1538207779:84 , srv1543687741:84 , srv1608357750:84 , srv1810958243:84 , srv1818438161:84 , srv1855907881:84 , srv1864565860:84 , srv1911192861:84 , srv1920347198:84 , srv1957063474:84 , srv2049343245:84 , srv2075945332:84 , srv208725600:84 , srv2098843663:84 , srv223856161:84 , srv277506339:84 , srv442262948:84 , srv450387239:84 , srv452572441:84 , srv473896525:84 , srv474635431:84 , srv48097626:84 , srv532846838:84 , srv535839589:84 , srv554471856:84 , srv581803984:84 , srv58335656:84 , srv615374544:84 , srv617419927:84 , srv620694243:84 , srv6275769:84 , srv66435335:84 , srv666726558:84 , srv674949784:84 , srv707343117:84 , srv722283793:84 , srv818328858:84 , srv869431368:84 , srv884720679:84 , srv896189683:84 , srv902522733:84 , srv10101954:85 , srv1138111820:85 , srv1175109797:85 , srv1213040339:85 , srv1233718715:85 , srv1350107038:85 , srv1359291857:85 , srv1379203835:85 , srv1399927783:85 , srv1442054582:85 , srv1470818325:85 , srv1644682694:85 , srv1708265297:85 , srv1729870646:85 , srv1821581544:85 , srv1841847373:85 , srv1866234766:85 , srv1913791995:85 , srv1944493647:85 , srv2045559203:85 , srv2084016653:85 , srv221960764:85 , srv267658464:85 , srv475090900:85 , srv512649340:85 , srv58126619:85 , srv595814832:85 , srv608568418:85 , srv628358055:85 , srv645769324:85 , srv928669290:85 , srv99140737:85 , srv1102901495:86 , srv1225231914:86 , srv312457303:86 , srv488763520:86 , srv531474607:86 , srv551360278:86 , srv662222376:86 , srv728373151:86 , srv898457340:86 , srv755815885:637 } 2023-07-18 05:53:14,401 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 257), ProcessCount=174 (was 174), AvailableMemoryMB=6763 (was 7524) 2023-07-18 05:53:14,410 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=252, ProcessCount=174, AvailableMemoryMB=6762 2023-07-18 05:53:14,411 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 05:53:14,411 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-18 05:53:14,422 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1195601244:84 , srv133468443:84 , srv1356392132:84 , srv1387940189:84 , srv1394717696:84 , srv1470398094:84 , srv1547086147:84 , srv1557842374:84 , srv159782938:84 , srv1652148547:84 , srv1918396897:84 , srv193895875:84 , srv1956016618:84 , srv2012394224:84 , srv2071668050:84 , srv211276241:84 , srv2145930832:84 , srv22863890:84 , srv250874248:84 , srv255709539:84 , srv26329858:84 , srv288758499:84 , srv318773669:84 , srv447378348:84 , srv671564884:84 , srv699589725:84 , srv758736582:84 , srv772194365:84 , srv883032482:84 , srv1505932085:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-18 05:53:14,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:53:14,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1547086147=7, srv447378348=24, srv211276241=16, srv772194365=28, srv159782938=9, srv1470398094=5, srv288758499=22, srv193895875=12, srv133468443=1, srv1652148547=10, srv318773669=23, srv2071668050=15, srv2145930832=17, srv1505932085=6, srv2012394224=14, srv1557842374=8, srv250874248=19, srv255709539=20, srv758736582=27, srv1394717696=4, srv1918396897=11, srv26329858=21, srv1195601244=0, srv699589725=26, srv1387940189=3, srv1956016618=13, srv671564884=25, srv1356392132=2, srv883032482=29, srv22863890=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:53:14,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:53:14,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:53:14,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:53:14,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:53:14,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:53:14,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:53:14,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-18 05:53:14,465 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.105804584984362, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:53:14,466 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912526430056727, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.105804584984362, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-18 05:55:14,444 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6177858 different iterations. Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912526430056727 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:55:14,451 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1195601244:90 , srv133468443:90 , srv1356392132:90 , srv1387940189:90 , srv1394717696:90 , srv1470398094:90 , srv1505932085:90 , srv1547086147:90 , srv1557842374:90 , srv159782938:90 , srv1652148547:90 , srv1918396897:90 , srv193895875:90 , srv1956016618:90 , srv2012394224:90 , srv2071668050:90 , srv211276241:90 , srv2145930832:90 , srv22863890:90 , srv250874248:90 , srv255709539:90 , srv26329858:90 , srv288758499:90 , srv318773669:90 , srv447378348:90 , srv671564884:90 , srv699589725:90 , srv758736582:90 , srv772194365:90 , srv883032482:90 } 2023-07-18 05:55:14,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 05:55:14,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1547086147=7, srv447378348=24, srv211276241=16, srv772194365=28, srv159782938=9, srv1470398094=5, srv288758499=22, srv193895875=12, srv133468443=1, srv1652148547=10, srv318773669=23, srv2071668050=15, srv2145930832=17, srv1505932085=6, srv2012394224=14, srv1557842374=8, srv250874248=19, srv255709539=20, srv758736582=27, srv1394717696=4, srv1918396897=11, srv26329858=21, srv1195601244=0, srv699589725=26, srv1387940189=3, srv1956016618=13, srv671564884=25, srv1356392132=2, srv883032482=29, srv22863890=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 05:55:14,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 05:55:14,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 05:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-18 05:55:14,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 05:55:14,472 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1195601244:90 , srv133468443:90 , srv1356392132:90 , srv1387940189:90 , srv1394717696:90 , srv1470398094:90 , srv1505932085:90 , srv1547086147:90 , srv1557842374:90 , srv159782938:90 , srv1652148547:90 , srv1918396897:90 , srv193895875:90 , srv1956016618:90 , srv2012394224:90 , srv2071668050:90 , srv211276241:90 , srv2145930832:90 , srv22863890:90 , srv250874248:90 , srv255709539:90 , srv26329858:90 , srv288758499:90 , srv318773669:90 , srv447378348:90 , srv671564884:90 , srv699589725:90 , srv758736582:90 , srv772194365:90 , srv883032482:90 } 2023-07-18 05:55:14,486 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=166 (was 252), ProcessCount=170 (was 174), AvailableMemoryMB=5831 (was 6762)