2024-01-20 06:53:14,417 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-20 06:53:14,756 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-20 06:53:14,866 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-20 06:53:14,937 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-20 06:53:14,946 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-20 06:53:14,962 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=170, AvailableMemoryMB=12699 2024-01-20 06:53:14,965 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-20 06:53:14,965 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-20 06:53:15,091 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1025717384:84 , srv1029167439:84 , srv1032343997:84 , srv1053318591:84 , srv1057074770:84 , srv1082879418:84 , srv1086261524:84 , srv1114347689:84 , srv1120563810:84 , srv1165998059:84 , srv1199452141:84 , srv1240871899:84 , srv124522882:84 , srv1257068844:84 , srv1261589652:84 , srv1262587797:84 , srv1280254014:84 , srv129688977:84 , srv1333553939:84 , srv1415788223:84 , srv1423360281:84 , srv1442392308:84 , srv1451707539:84 , srv1463015869:84 , srv1475816151:84 , srv1492119409:84 , srv150537713:84 , srv1517151350:84 , srv1543320897:84 , srv1592784928:84 , srv1622679927:84 , srv1645485186:84 , srv165408307:84 , srv167172825:84 , srv1725516582:84 , srv1745212745:84 , srv1746434449:84 , srv1754651078:84 , srv1788463766:84 , srv179556139:84 , srv1809071829:84 , srv1830157404:84 , srv1841714523:84 , srv185130361:84 , srv1853833944:84 , srv1860567309:84 , srv1871535660:84 , srv1890872381:84 , srv1896221672:84 , srv1907929901:84 , srv1914818360:84 , srv1917776452:84 , srv1953351370:84 , srv1977618499:84 , srv1990635128:84 , srv1997158064:84 , srv203318124:84 , srv2054007192:84 , srv2114627352:84 , srv27317235:84 , srv289004830:84 , srv292895183:84 , srv303698750:84 , srv30716138:84 , srv33062904:84 , srv364259669:84 , srv371884579:84 , srv405164872:84 , srv426648286:84 , srv454443466:84 , srv479533716:84 , srv48958104:84 , srv529120931:84 , srv539993405:84 , srv547084580:84 , srv607796383:84 , srv625811831:84 , srv626403266:84 , srv627051837:84 , srv636932597:84 , srv673640970:84 , srv718151460:84 , srv724309063:84 , srv732167774:84 , srv800316077:84 , srv82367850:84 , srv840865752:84 , srv845084786:84 , srv860684219:84 , srv867264155:84 , srv887472180:84 , srv895457844:84 , srv908497976:84 , srv917627301:84 , srv926169697:84 , srv932528799:84 , srv985645110:84 , srv98830528:84 , srv995187217:84 , srv39551570:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-20 06:53:15,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:15,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167172825=33, srv718151460=82, srv364259669=65, srv539993405=74, srv1086261524=6, srv1853833944=44, srv1860567309=45, srv1645485186=31, srv48958104=72, srv1463015869=23, srv27317235=59, srv732167774=84, srv124522882=12, srv985645110=97, srv673640970=81, srv1592784928=29, srv1280254014=16, srv917627301=94, srv1053318591=3, srv800316077=85, srv1029167439=1, srv860684219=89, srv1809071829=40, srv289004830=60, srv908497976=93, srv1997158064=55, srv2114627352=58, srv1871535660=46, srv33062904=64, srv1120563810=8, srv1032343997=2, srv292895183=61, srv887472180=91, srv547084580=75, srv1914818360=50, srv840865752=87, srv895457844=92, srv1622679927=30, srv1990635128=54, srv1114347689=7, srv845084786=88, srv724309063=83, srv303698750=62, srv30716138=63, srv932528799=96, srv1953351370=52, srv165408307=32, srv185130361=43, srv995187217=99, srv1475816151=24, srv1746434449=36, srv636932597=80, srv1423360281=20, srv1442392308=21, srv1543320897=28, srv1451707539=22, srv625811831=77, srv1261589652=14, srv405164872=68, srv867264155=90, srv1788463766=38, srv1754651078=37, srv1025717384=0, srv426648286=69, srv1907929901=49, srv454443466=70, srv1165998059=9, srv1517151350=27, srv1841714523=42, srv98830528=98, srv150537713=26, srv129688977=17, srv529120931=73, srv1240871899=11, srv1415788223=19, srv627051837=79, srv479533716=71, srv371884579=66, srv1890872381=47, srv1745212745=35, srv179556139=39, srv626403266=78, srv1262587797=15, srv2054007192=57, srv1830157404=41, srv1199452141=10, srv1257068844=13, srv82367850=86, srv1977618499=53, srv1333553939=18, srv1492119409=25, srv203318124=56, srv1725516582=34, srv607796383=76, srv39551570=67, srv1917776452=51, srv1057074770=4, srv1896221672=48, srv1082879418=5, srv926169697=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:15,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-20 06:53:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-20 06:53:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-20 06:53:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-20 06:53:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-20 06:53:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-20 06:53:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-20 06:53:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-20 06:53:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-20 06:53:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-20 06:53:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-20 06:53:15,548 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:15,548 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-20 06:53:15,549 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-20 06:53:15,974 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 760 ms to try 5000 different iterations. Found a solution that moves 4720 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17688976815635077. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038946085884033255, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056088354221146594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038946085884033255, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23708333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:15,993 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv150537713:81 , srv1114347689:82 , srv1853833944:82 , srv917627301:82 , srv1165998059:83 , srv1592784928:83 , srv1645485186:83 , srv1725516582:83 , srv1890872381:83 , srv1990635128:83 , srv292895183:83 , srv426648286:83 , srv926169697:83 , srv129688977:84 , srv1463015869:84 , srv167172825:84 , srv1841714523:84 , srv33062904:84 , srv364259669:84 , srv732167774:84 , srv1029167439:85 , srv1240871899:85 , srv1257068844:85 , srv1262587797:85 , srv1423360281:85 , srv1745212745:85 , srv1746434449:85 , srv179556139:85 , srv185130361:85 , srv1871535660:85 , srv405164872:85 , srv607796383:85 , srv626403266:85 , srv673640970:85 , srv724309063:85 , srv840865752:85 , srv895457844:85 , srv908497976:85 , srv995187217:85 , srv1082879418:86 , srv1199452141:86 , srv1415788223:86 , srv1451707539:86 , srv1475816151:86 , srv1754651078:86 , srv1809071829:86 , srv1830157404:86 , srv1907929901:86 , srv1977618499:86 , srv203318124:86 , srv2054007192:86 , srv303698750:86 , srv454443466:86 , srv529120931:86 , srv539993405:86 , srv1053318591:87 , srv1120563810:87 , srv165408307:87 , srv1896221672:87 , srv1917776452:87 , srv27317235:87 , srv30716138:87 , srv627051837:87 , srv800316077:87 , srv82367850:87 , srv985645110:87 , srv1032343997:88 , srv1057074770:88 , srv1261589652:88 , srv1280254014:88 , srv1442392308:88 , srv1622679927:88 , srv1788463766:88 , srv1953351370:88 , srv1997158064:88 , srv289004830:88 , srv371884579:88 , srv479533716:88 , srv1025717384:89 , srv1086261524:89 , srv124522882:89 , srv1543320897:89 , srv625811831:89 , srv636932597:89 , srv867264155:89 , srv887472180:89 , srv98830528:89 , srv48958104:90 , srv547084580:90 , srv718151460:90 , srv845084786:90 , srv932528799:90 , srv1860567309:91 , srv1914818360:91 , srv860684219:91 , srv1492119409:92 , srv2114627352:92 , srv1517151350:94 , srv1333553939:96 , srv39551570:436 } 2024-01-20 06:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167172825=33, srv718151460=82, srv364259669=65, srv539993405=74, srv1086261524=6, srv1853833944=44, srv1860567309=45, srv1645485186=31, srv48958104=72, srv1463015869=23, srv27317235=59, srv732167774=84, srv124522882=12, srv985645110=97, srv673640970=81, srv1592784928=29, srv1280254014=16, srv917627301=94, srv1053318591=3, srv800316077=85, srv1029167439=1, srv860684219=89, srv1809071829=40, srv289004830=60, srv908497976=93, srv1997158064=55, srv2114627352=58, srv1871535660=46, srv33062904=64, srv1120563810=8, srv1032343997=2, srv292895183=61, srv887472180=91, srv547084580=75, srv1914818360=50, srv840865752=87, srv895457844=92, srv1622679927=30, srv1990635128=54, srv1114347689=7, srv845084786=88, srv724309063=83, srv303698750=62, srv30716138=63, srv932528799=96, srv1953351370=52, srv165408307=32, srv185130361=43, srv995187217=99, srv1475816151=24, srv1746434449=36, srv636932597=80, srv1423360281=20, srv1442392308=21, srv1543320897=28, srv1451707539=22, srv625811831=77, srv1261589652=14, srv405164872=68, srv867264155=90, srv1788463766=38, srv1754651078=37, srv1025717384=0, srv426648286=69, srv1907929901=49, srv454443466=70, srv1165998059=9, srv1517151350=27, srv1841714523=42, srv98830528=98, srv150537713=26, srv129688977=17, srv529120931=73, srv1240871899=11, srv1415788223=19, srv627051837=79, srv479533716=71, srv371884579=66, srv1890872381=47, srv1745212745=35, srv179556139=39, srv626403266=78, srv1262587797=15, srv2054007192=57, srv1830157404=41, srv1199452141=10, srv1257068844=13, srv82367850=86, srv1977618499=53, srv1333553939=18, srv1492119409=25, srv203318124=56, srv1725516582=34, srv607796383=76, srv39551570=67, srv1917776452=51, srv1057074770=4, srv1896221672=48, srv1082879418=5, srv926169697=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:16,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:16,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:16,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:16,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:16,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:16,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:16,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-20 06:53:16,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-20 06:53:16,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-20 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-20 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-20 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-20 06:53:16,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-20 06:53:16,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-20 06:53:16,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-20 06:53:16,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-20 06:53:16,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-20 06:53:16,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-20 06:53:16,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-20 06:53:16,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-20 06:53:16,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-20 06:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-20 06:53:16,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-20 06:53:16,203 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038946085884033255, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056088354221146594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038946085884033255, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23708333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:16,204 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-20 06:53:16,204 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17685671800995584, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038946085884033255, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056088354221146594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038946085884033255, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23708333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-20 06:53:16,524 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 445 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.17685671800995584 to a new imbalance of 0.014948827459199385. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022037956577595268); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03329964627739724, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022037956577595268); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043833333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:16,533 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1333553939:75 , srv1517151350:77 , srv1492119409:80 , srv1860567309:80 , srv1914818360:81 , srv2114627352:81 , srv1057074770:82 , srv48958104:82 , srv636932597:82 , srv860684219:82 , srv98830528:82 , srv1025717384:83 , srv1086261524:83 , srv1442392308:83 , srv1896221672:83 , srv1997158064:83 , srv371884579:83 , srv479533716:83 , srv547084580:83 , srv625811831:83 , srv845084786:83 , srv932528799:83 , srv1053318591:84 , srv1543320897:84 , srv1622679927:84 , srv1754651078:84 , srv1788463766:84 , srv1953351370:84 , srv1977618499:84 , srv627051837:84 , srv718151460:84 , srv867264155:84 , srv985645110:84 , srv1029167439:85 , srv1032343997:85 , srv1120563810:85 , srv1199452141:85 , srv1261589652:85 , srv1907929901:85 , srv1917776452:85 , srv203318124:85 , srv27317235:85 , srv289004830:85 , srv82367850:85 , srv887472180:85 , srv124522882:86 , srv1262587797:86 , srv1280254014:86 , srv1451707539:86 , srv1475816151:86 , srv165408307:86 , srv1745212745:86 , srv1830157404:86 , srv1841714523:86 , srv1871535660:86 , srv2054007192:86 , srv30716138:86 , srv364259669:86 , srv539993405:86 , srv626403266:86 , srv724309063:86 , srv732167774:86 , srv895457844:86 , srv908497976:86 , srv1257068844:87 , srv1415788223:87 , srv1423360281:87 , srv1463015869:87 , srv1592784928:87 , srv1809071829:87 , srv185130361:87 , srv303698750:87 , srv405164872:87 , srv454443466:87 , srv607796383:87 , srv800316077:87 , srv840865752:87 , srv1082879418:88 , srv1240871899:88 , srv129688977:88 , srv1645485186:88 , srv167172825:88 , srv1746434449:88 , srv1890872381:88 , srv292895183:88 , srv33062904:88 , srv673640970:88 , srv995187217:88 , srv1165998059:89 , srv179556139:89 , srv426648286:89 , srv1990635128:90 , srv529120931:90 , srv917627301:90 , srv1114347689:91 , srv150537713:91 , srv1725516582:91 , srv1853833944:91 , srv926169697:91 , srv39551570:534 } 2024-01-20 06:53:16,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:16,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167172825=33, srv718151460=82, srv364259669=65, srv539993405=74, srv1086261524=6, srv1853833944=44, srv1860567309=45, srv1645485186=31, srv48958104=72, srv1463015869=23, srv27317235=59, srv732167774=84, srv124522882=12, srv985645110=97, srv673640970=81, srv1592784928=29, srv1280254014=16, srv917627301=94, srv1053318591=3, srv800316077=85, srv1029167439=1, srv860684219=89, srv1809071829=40, srv289004830=60, srv908497976=93, srv1997158064=55, srv2114627352=58, srv1871535660=46, srv33062904=64, srv1120563810=8, srv1032343997=2, srv292895183=61, srv887472180=91, srv547084580=75, srv1914818360=50, srv840865752=87, srv895457844=92, srv1622679927=30, srv1990635128=54, srv1114347689=7, srv845084786=88, srv724309063=83, srv303698750=62, srv30716138=63, srv932528799=96, srv1953351370=52, srv165408307=32, srv185130361=43, srv995187217=99, srv1475816151=24, srv1746434449=36, srv636932597=80, srv1423360281=20, srv1442392308=21, srv1543320897=28, srv1451707539=22, srv625811831=77, srv1261589652=14, srv405164872=68, srv867264155=90, srv1788463766=38, srv1754651078=37, srv1025717384=0, srv426648286=69, srv1907929901=49, srv454443466=70, srv1165998059=9, srv1517151350=27, srv1841714523=42, srv98830528=98, srv150537713=26, srv129688977=17, srv529120931=73, srv1240871899=11, srv1415788223=19, srv627051837=79, srv479533716=71, srv371884579=66, srv1890872381=47, srv1745212745=35, srv179556139=39, srv626403266=78, srv1262587797=15, srv2054007192=57, srv1830157404=41, srv1199452141=10, srv1257068844=13, srv82367850=86, srv1977618499=53, srv1333553939=18, srv1492119409=25, srv203318124=56, srv1725516582=34, srv607796383=76, srv39551570=67, srv1917776452=51, srv1057074770=4, srv1896221672=48, srv1082879418=5, srv926169697=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-20 06:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-20 06:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-20 06:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-20 06:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-20 06:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-20 06:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-20 06:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-20 06:53:16,678 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022037956577595268); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03329964627739724, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022037956577595268); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043833333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:16,679 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-20 06:53:16,679 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014930544846437266, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022037956577595268); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03329964627739724, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022037956577595268); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043833333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-20 06:53:16,929 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 347 ms to try 5000 different iterations. Found a solution that moves 999 regions; Going from a computed imbalance of 0.014930544846437266 to a new imbalance of 8.974034789007219E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005661173367839546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005661173367839546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:16,930 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv529120931:82 , srv1082879418:83 , srv124522882:83 , srv1725516582:83 , srv1990635128:83 , srv800316077:83 , srv887472180:83 , srv926169697:83 , srv1032343997:84 , srv1114347689:84 , srv1240871899:84 , srv1280254014:84 , srv1415788223:84 , srv1451707539:84 , srv179556139:84 , srv1853833944:84 , srv2114627352:84 , srv289004830:84 , srv303698750:84 , srv454443466:84 , srv718151460:84 , srv845084786:84 , srv995187217:84 , srv1165998059:85 , srv1257068844:85 , srv1261589652:85 , srv150537713:85 , srv1543320897:85 , srv165408307:85 , srv167172825:85 , srv1788463766:85 , srv1809071829:85 , srv1830157404:85 , srv1917776452:85 , srv1953351370:85 , srv1997158064:85 , srv27317235:85 , srv30716138:85 , srv33062904:85 , srv405164872:85 , srv426648286:85 , srv539993405:85 , srv547084580:85 , srv607796383:85 , srv625811831:85 , srv673640970:85 , srv840865752:85 , srv860684219:85 , srv867264155:85 , srv932528799:85 , srv1025717384:86 , srv1053318591:86 , srv1086261524:86 , srv1120563810:86 , srv1262587797:86 , srv129688977:86 , srv1423360281:86 , srv1442392308:86 , srv1463015869:86 , srv1475816151:86 , srv1492119409:86 , srv1517151350:86 , srv1622679927:86 , srv185130361:86 , srv1860567309:86 , srv1890872381:86 , srv1907929901:86 , srv203318124:86 , srv2054007192:86 , srv292895183:86 , srv371884579:86 , srv479533716:86 , srv48958104:86 , srv626403266:86 , srv627051837:86 , srv636932597:86 , srv724309063:86 , srv82367850:86 , srv908497976:86 , srv917627301:86 , srv1199452141:87 , srv1645485186:87 , srv1745212745:87 , srv1746434449:87 , srv1871535660:87 , srv1914818360:87 , srv732167774:87 , srv895457844:87 , srv98830528:87 , srv1029167439:88 , srv1057074770:88 , srv1333553939:88 , srv1592784928:88 , srv1754651078:88 , srv1841714523:88 , srv1977618499:88 , srv364259669:88 , srv985645110:88 , srv1896221672:89 , srv39551570:538 } 2024-01-20 06:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167172825=33, srv718151460=82, srv364259669=65, srv539993405=74, srv1086261524=6, srv1853833944=44, srv1860567309=45, srv1645485186=31, srv48958104=72, srv1463015869=23, srv27317235=59, srv732167774=84, srv124522882=12, srv985645110=97, srv673640970=81, srv1592784928=29, srv1280254014=16, srv917627301=94, srv1053318591=3, srv800316077=85, srv1029167439=1, srv860684219=89, srv1809071829=40, srv289004830=60, srv908497976=93, srv1997158064=55, srv2114627352=58, srv1871535660=46, srv33062904=64, srv1120563810=8, srv1032343997=2, srv292895183=61, srv887472180=91, srv547084580=75, srv1914818360=50, srv840865752=87, srv895457844=92, srv1622679927=30, srv1990635128=54, srv1114347689=7, srv845084786=88, srv724309063=83, srv303698750=62, srv30716138=63, srv932528799=96, srv1953351370=52, srv165408307=32, srv185130361=43, srv995187217=99, srv1475816151=24, srv1746434449=36, srv636932597=80, srv1423360281=20, srv1442392308=21, srv1543320897=28, srv1451707539=22, srv625811831=77, srv1261589652=14, srv405164872=68, srv867264155=90, srv1788463766=38, srv1754651078=37, srv1025717384=0, srv426648286=69, srv1907929901=49, srv454443466=70, srv1165998059=9, srv1517151350=27, srv1841714523=42, srv98830528=98, srv150537713=26, srv129688977=17, srv529120931=73, srv1240871899=11, srv1415788223=19, srv627051837=79, srv479533716=71, srv371884579=66, srv1890872381=47, srv1745212745=35, srv179556139=39, srv626403266=78, srv1262587797=15, srv2054007192=57, srv1830157404=41, srv1199452141=10, srv1257068844=13, srv82367850=86, srv1977618499=53, srv1333553939=18, srv1492119409=25, srv203318124=56, srv1725516582=34, srv607796383=76, srv39551570=67, srv1917776452=51, srv1057074770=4, srv1896221672=48, srv1082879418=5, srv926169697=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-20 06:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-20 06:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-20 06:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-20 06:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-20 06:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-20 06:53:17,064 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005661173367839546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005661173367839546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:17,064 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-20 06:53:17,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.904083313904363E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005661173367839546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005661173367839546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-20 06:53:17,432 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 469 ms to try 5000 different iterations. Found a solution that moves 229 regions; Going from a computed imbalance of 8.904083313904363E-4 to a new imbalance of 3.674843967793885E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.025444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:17,433 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1746434449:82 , srv1896221672:82 , srv1333553939:83 , srv1645485186:83 , srv179556139:83 , srv1914818360:83 , srv895457844:83 , srv985645110:83 , srv1025717384:84 , srv1029167439:84 , srv1053318591:84 , srv1057074770:84 , srv1086261524:84 , srv1114347689:84 , srv1199452141:84 , srv124522882:84 , srv1261589652:84 , srv1262587797:84 , srv1280254014:84 , srv129688977:84 , srv1423360281:84 , srv1475816151:84 , srv1543320897:84 , srv1592784928:84 , srv1622679927:84 , srv165408307:84 , srv167172825:84 , srv1745212745:84 , srv1754651078:84 , srv1809071829:84 , srv1841714523:84 , srv185130361:84 , srv1871535660:84 , srv1977618499:84 , srv2054007192:84 , srv30716138:84 , srv364259669:84 , srv48958104:84 , srv529120931:84 , srv547084580:84 , srv627051837:84 , srv673640970:84 , srv718151460:84 , srv724309063:84 , srv860684219:84 , srv867264155:84 , srv917627301:84 , srv98830528:84 , srv1032343997:85 , srv1082879418:85 , srv1120563810:85 , srv1165998059:85 , srv1240871899:85 , srv1257068844:85 , srv1415788223:85 , srv1442392308:85 , srv1463015869:85 , srv1492119409:85 , srv1517151350:85 , srv1725516582:85 , srv1788463766:85 , srv1830157404:85 , srv1853833944:85 , srv1890872381:85 , srv1917776452:85 , srv1953351370:85 , srv203318124:85 , srv2114627352:85 , srv27317235:85 , srv289004830:85 , srv292895183:85 , srv303698750:85 , srv33062904:85 , srv371884579:85 , srv405164872:85 , srv479533716:85 , srv539993405:85 , srv607796383:85 , srv625811831:85 , srv626403266:85 , srv636932597:85 , srv732167774:85 , srv800316077:85 , srv82367850:85 , srv840865752:85 , srv845084786:85 , srv887472180:85 , srv926169697:85 , srv932528799:85 , srv995187217:85 , srv1451707539:86 , srv150537713:86 , srv1907929901:86 , srv1990635128:86 , srv1997158064:86 , srv426648286:86 , srv454443466:86 , srv908497976:86 , srv1860567309:87 , srv39551570:633 } 2024-01-20 06:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:17,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167172825=33, srv718151460=82, srv364259669=65, srv539993405=74, srv1086261524=6, srv1853833944=44, srv1860567309=45, srv1645485186=31, srv48958104=72, srv1463015869=23, srv27317235=59, srv732167774=84, srv124522882=12, srv985645110=97, srv673640970=81, srv1592784928=29, srv1280254014=16, srv917627301=94, srv1053318591=3, srv800316077=85, srv1029167439=1, srv860684219=89, srv1809071829=40, srv289004830=60, srv908497976=93, srv1997158064=55, srv2114627352=58, srv1871535660=46, srv33062904=64, srv1120563810=8, srv1032343997=2, srv292895183=61, srv887472180=91, srv547084580=75, srv1914818360=50, srv840865752=87, srv895457844=92, srv1622679927=30, srv1990635128=54, srv1114347689=7, srv845084786=88, srv724309063=83, srv303698750=62, srv30716138=63, srv932528799=96, srv1953351370=52, srv165408307=32, srv185130361=43, srv995187217=99, srv1475816151=24, srv1746434449=36, srv636932597=80, srv1423360281=20, srv1442392308=21, srv1543320897=28, srv1451707539=22, srv625811831=77, srv1261589652=14, srv405164872=68, srv867264155=90, srv1788463766=38, srv1754651078=37, srv1025717384=0, srv426648286=69, srv1907929901=49, srv454443466=70, srv1165998059=9, srv1517151350=27, srv1841714523=42, srv98830528=98, srv150537713=26, srv129688977=17, srv529120931=73, srv1240871899=11, srv1415788223=19, srv627051837=79, srv479533716=71, srv371884579=66, srv1890872381=47, srv1745212745=35, srv179556139=39, srv626403266=78, srv1262587797=15, srv2054007192=57, srv1830157404=41, srv1199452141=10, srv1257068844=13, srv82367850=86, srv1977618499=53, srv1333553939=18, srv1492119409=25, srv203318124=56, srv1725516582=34, srv607796383=76, srv39551570=67, srv1917776452=51, srv1057074770=4, srv1896221672=48, srv1082879418=5, srv926169697=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:17,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:17,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-20 06:53:17,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-20 06:53:17,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-20 06:53:17,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-20 06:53:17,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-20 06:53:17,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-20 06:53:17,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-20 06:53:17,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-20 06:53:17,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-20 06:53:17,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-20 06:53:17,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-20 06:53:17,563 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:17,563 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-20 06:53:17,564 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.51449474058113E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-20 06:53:17,821 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 352 ms to try 5000 different iterations. Found a solution that moves 41 regions; Going from a computed imbalance of 3.51449474058113E-5 to a new imbalance of 6.6850738083274685E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.004555555555555556); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:17,821 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1860567309:82 , srv1120563810:83 , srv1492119409:83 , srv150537713:83 , srv1907929901:83 , srv426648286:83 , srv454443466:83 , srv82367850:83 , srv908497976:83 , srv932528799:83 , srv1025717384:84 , srv1029167439:84 , srv1032343997:84 , srv1057074770:84 , srv1082879418:84 , srv1086261524:84 , srv1165998059:84 , srv1199452141:84 , srv1240871899:84 , srv124522882:84 , srv1257068844:84 , srv1261589652:84 , srv1280254014:84 , srv129688977:84 , srv1333553939:84 , srv1415788223:84 , srv1423360281:84 , srv1442392308:84 , srv1451707539:84 , srv1463015869:84 , srv1475816151:84 , srv1517151350:84 , srv1543320897:84 , srv1592784928:84 , srv1622679927:84 , srv165408307:84 , srv1725516582:84 , srv1745212745:84 , srv1746434449:84 , srv1754651078:84 , srv1788463766:84 , srv1809071829:84 , srv1830157404:84 , srv1841714523:84 , srv185130361:84 , srv1853833944:84 , srv1871535660:84 , srv1890872381:84 , srv1914818360:84 , srv1917776452:84 , srv1953351370:84 , srv1977618499:84 , srv1990635128:84 , srv1997158064:84 , srv203318124:84 , srv2054007192:84 , srv2114627352:84 , srv27317235:84 , srv289004830:84 , srv292895183:84 , srv303698750:84 , srv30716138:84 , srv33062904:84 , srv364259669:84 , srv371884579:84 , srv405164872:84 , srv479533716:84 , srv48958104:84 , srv529120931:84 , srv539993405:84 , srv547084580:84 , srv607796383:84 , srv625811831:84 , srv626403266:84 , srv636932597:84 , srv673640970:84 , srv718151460:84 , srv732167774:84 , srv800316077:84 , srv840865752:84 , srv845084786:84 , srv860684219:84 , srv867264155:84 , srv887472180:84 , srv917627301:84 , srv926169697:84 , srv985645110:84 , srv98830528:84 , srv995187217:84 , srv1053318591:85 , srv1114347689:85 , srv1262587797:85 , srv1645485186:85 , srv167172825:85 , srv179556139:85 , srv1896221672:85 , srv627051837:85 , srv724309063:85 , srv895457844:85 , srv39551570:685 } 2024-01-20 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167172825=33, srv718151460=82, srv364259669=65, srv539993405=74, srv1086261524=6, srv1853833944=44, srv1860567309=45, srv1645485186=31, srv48958104=72, srv1463015869=23, srv27317235=59, srv732167774=84, srv124522882=12, srv985645110=97, srv673640970=81, srv1592784928=29, srv1280254014=16, srv917627301=94, srv1053318591=3, srv800316077=85, srv1029167439=1, srv860684219=89, srv1809071829=40, srv289004830=60, srv908497976=93, srv1997158064=55, srv2114627352=58, srv1871535660=46, srv33062904=64, srv1120563810=8, srv1032343997=2, srv292895183=61, srv887472180=91, srv547084580=75, srv1914818360=50, srv840865752=87, srv895457844=92, srv1622679927=30, srv1990635128=54, srv1114347689=7, srv845084786=88, srv724309063=83, srv303698750=62, srv30716138=63, srv932528799=96, srv1953351370=52, srv165408307=32, srv185130361=43, srv995187217=99, srv1475816151=24, srv1746434449=36, srv636932597=80, srv1423360281=20, srv1442392308=21, srv1543320897=28, srv1451707539=22, srv625811831=77, srv1261589652=14, srv405164872=68, srv867264155=90, srv1788463766=38, srv1754651078=37, srv1025717384=0, srv426648286=69, srv1907929901=49, srv454443466=70, srv1165998059=9, srv1517151350=27, srv1841714523=42, srv98830528=98, srv150537713=26, srv129688977=17, srv529120931=73, srv1240871899=11, srv1415788223=19, srv627051837=79, srv479533716=71, srv371884579=66, srv1890872381=47, srv1745212745=35, srv179556139=39, srv626403266=78, srv1262587797=15, srv2054007192=57, srv1830157404=41, srv1199452141=10, srv1257068844=13, srv82367850=86, srv1977618499=53, srv1333553939=18, srv1492119409=25, srv203318124=56, srv1725516582=34, srv607796383=76, srv39551570=67, srv1917776452=51, srv1057074770=4, srv1896221672=48, srv1082879418=5, srv926169697=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:17,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:17,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:17,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:17,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-20 06:53:17,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-20 06:53:17,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-20 06:53:17,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-20 06:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-20 06:53:17,952 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:17,952 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1860567309:82 , srv1120563810:83 , srv1492119409:83 , srv150537713:83 , srv1907929901:83 , srv426648286:83 , srv454443466:83 , srv82367850:83 , srv908497976:83 , srv932528799:83 , srv1025717384:84 , srv1029167439:84 , srv1032343997:84 , srv1057074770:84 , srv1082879418:84 , srv1086261524:84 , srv1165998059:84 , srv1199452141:84 , srv1240871899:84 , srv124522882:84 , srv1257068844:84 , srv1261589652:84 , srv1280254014:84 , srv129688977:84 , srv1333553939:84 , srv1415788223:84 , srv1423360281:84 , srv1442392308:84 , srv1451707539:84 , srv1463015869:84 , srv1475816151:84 , srv1517151350:84 , srv1543320897:84 , srv1592784928:84 , srv1622679927:84 , srv165408307:84 , srv1725516582:84 , srv1745212745:84 , srv1746434449:84 , srv1754651078:84 , srv1788463766:84 , srv1809071829:84 , srv1830157404:84 , srv1841714523:84 , srv185130361:84 , srv1853833944:84 , srv1871535660:84 , srv1890872381:84 , srv1914818360:84 , srv1917776452:84 , srv1953351370:84 , srv1977618499:84 , srv1990635128:84 , srv1997158064:84 , srv203318124:84 , srv2054007192:84 , srv2114627352:84 , srv27317235:84 , srv289004830:84 , srv292895183:84 , srv303698750:84 , srv30716138:84 , srv33062904:84 , srv364259669:84 , srv371884579:84 , srv405164872:84 , srv479533716:84 , srv48958104:84 , srv529120931:84 , srv539993405:84 , srv547084580:84 , srv607796383:84 , srv625811831:84 , srv626403266:84 , srv636932597:84 , srv673640970:84 , srv718151460:84 , srv732167774:84 , srv800316077:84 , srv840865752:84 , srv845084786:84 , srv860684219:84 , srv867264155:84 , srv887472180:84 , srv917627301:84 , srv926169697:84 , srv985645110:84 , srv98830528:84 , srv995187217:84 , srv1053318591:85 , srv1114347689:85 , srv1262587797:85 , srv1645485186:85 , srv167172825:85 , srv179556139:85 , srv1896221672:85 , srv627051837:85 , srv724309063:85 , srv895457844:85 , srv39551570:685 } 2024-01-20 06:53:17,978 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=293 (was 301), ProcessCount=170 (was 170), AvailableMemoryMB=12101 (was 12699) 2024-01-20 06:53:17,986 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=293, ProcessCount=170, AvailableMemoryMB=12100 2024-01-20 06:53:17,987 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-20 06:53:17,987 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-20 06:53:17,998 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1049696329:84 , srv1118724511:84 , srv1231123825:84 , srv1445840751:84 , srv1480241911:84 , srv1482046696:84 , srv1523307257:84 , srv154230903:84 , srv1593201745:84 , srv161477325:84 , srv16196374:84 , srv1733107696:84 , srv1755910727:84 , srv1806920011:84 , srv1811907237:84 , srv1910647035:84 , srv1917995479:84 , srv1931748769:84 , srv1955692069:84 , srv1966633613:84 , srv407489557:84 , srv418970085:84 , srv451147275:84 , srv458792336:84 , srv647778212:84 , srv693582993:84 , srv767970663:84 , srv910159449:84 , srv995120765:84 , srv1270326704:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-20 06:53:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:53:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1811907237=15, srv647778212=25, srv1049696329=0, srv767970663=27, srv1118724511=1, srv1231123825=2, srv1523307257=7, srv1806920011=14, srv1955692069=19, srv1480241911=5, srv995120765=29, srv1482046696=6, srv1755910727=13, srv1917995479=17, srv16196374=11, srv1733107696=12, srv418970085=22, srv407489557=21, srv910159449=28, srv1966633613=20, srv1270326704=3, srv1593201745=9, srv1931748769=18, srv458792336=24, srv693582993=26, srv161477325=10, srv154230903=8, srv1910647035=16, srv1445840751=4, srv451147275=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:53:18,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:53:18,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:53:18,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-20 06:53:18,027 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1244124786507972, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:53:18,028 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912585062922068, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1244124786507972, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-20 06:55:18,013 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5405886 different iterations. Found a solution that moves 1857 regions; Going from a computed imbalance of 0.9912585062922068 to a new imbalance of 4.334330639506329E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:55:18,021 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1049696329:90 , srv1118724511:90 , srv1231123825:90 , srv1270326704:90 , srv1445840751:90 , srv1480241911:90 , srv1482046696:90 , srv1523307257:90 , srv154230903:90 , srv1593201745:90 , srv161477325:90 , srv16196374:90 , srv1733107696:90 , srv1755910727:90 , srv1806920011:90 , srv1811907237:90 , srv1910647035:90 , srv1917995479:90 , srv1931748769:90 , srv1955692069:90 , srv1966633613:90 , srv407489557:90 , srv418970085:90 , srv451147275:90 , srv458792336:90 , srv647778212:90 , srv693582993:90 , srv767970663:90 , srv910159449:90 , srv995120765:90 } 2024-01-20 06:55:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-20 06:55:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1811907237=15, srv647778212=25, srv1049696329=0, srv767970663=27, srv1118724511=1, srv1231123825=2, srv1523307257=7, srv1806920011=14, srv1955692069=19, srv1480241911=5, srv995120765=29, srv1482046696=6, srv1755910727=13, srv1917995479=17, srv16196374=11, srv1733107696=12, srv418970085=22, srv407489557=21, srv910159449=28, srv1966633613=20, srv1270326704=3, srv1593201745=9, srv1931748769=18, srv458792336=24, srv693582993=26, srv161477325=10, srv154230903=8, srv1910647035=16, srv1445840751=4, srv451147275=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-20 06:55:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-20 06:55:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-20 06:55:18,040 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-20 06:55:18,041 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1049696329:90 , srv1118724511:90 , srv1231123825:90 , srv1270326704:90 , srv1445840751:90 , srv1480241911:90 , srv1482046696:90 , srv1523307257:90 , srv154230903:90 , srv1593201745:90 , srv161477325:90 , srv16196374:90 , srv1733107696:90 , srv1755910727:90 , srv1806920011:90 , srv1811907237:90 , srv1910647035:90 , srv1917995479:90 , srv1931748769:90 , srv1955692069:90 , srv1966633613:90 , srv407489557:90 , srv418970085:90 , srv451147275:90 , srv458792336:90 , srv647778212:90 , srv693582993:90 , srv767970663:90 , srv910159449:90 , srv995120765:90 } 2024-01-20 06:55:18,058 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=265 (was 293), ProcessCount=168 (was 170), AvailableMemoryMB=12253 (was 12100) - AvailableMemoryMB LEAK? -