2023-05-17 20:53:39,612 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-17 20:53:39,897 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-17 20:53:40,008 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-17 20:53:40,068 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-17 20:53:40,075 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-17 20:53:40,089 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=174, AvailableMemoryMB=4287 2023-05-17 20:53:40,090 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-17 20:53:40,091 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-17 20:53:40,253 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009167469:84 , srv1047413213:84 , srv1104562388:84 , srv1181358362:84 , srv1193360237:84 , srv1257032753:84 , srv1257352809:84 , srv1269778082:84 , srv1313699472:84 , srv1352302196:84 , srv1379050822:84 , srv142456192:84 , srv1442694669:84 , srv1450110954:84 , srv1480441065:84 , srv1496817440:84 , srv1512569711:84 , srv1549064741:84 , srv1550553316:84 , srv1569570377:84 , srv1597347682:84 , srv1612810700:84 , srv1644333093:84 , srv1670116464:84 , srv1677955006:84 , srv1678090744:84 , srv1695340801:84 , srv1725590265:84 , srv1757244962:84 , srv1765104397:84 , srv1785966590:84 , srv1797866175:84 , srv181058790:84 , srv1821370155:84 , srv1821803894:84 , srv1848095134:84 , srv1870036789:84 , srv1876191076:84 , srv1935774375:84 , srv1941182402:84 , srv1945110437:84 , srv1947750038:84 , srv1949320994:84 , srv1950069499:84 , srv2000347201:84 , srv2004452166:84 , srv2058648266:84 , srv2086927628:84 , srv2091932160:84 , srv2095749286:84 , srv2103564377:84 , srv211541025:84 , srv2117746073:84 , srv2119116659:84 , srv21354799:84 , srv25290698:84 , srv298008411:84 , srv3349383:84 , srv401006438:84 , srv423540818:84 , srv478785675:84 , srv493251843:84 , srv562554696:84 , srv569780:84 , srv574404471:84 , srv596472655:84 , srv617799203:84 , srv632067014:84 , srv632696262:84 , srv651929968:84 , srv66151263:84 , srv668426153:84 , srv670879095:84 , srv679204579:84 , srv679693484:84 , srv684108754:84 , srv707537188:84 , srv714015773:84 , srv726992922:84 , srv751541073:84 , srv767828861:84 , srv783205531:84 , srv833569513:84 , srv867556935:84 , srv869997298:84 , srv878673235:84 , srv878700085:84 , srv87974878:84 , srv8992972:84 , srv900800212:84 , srv908366295:84 , srv914657124:84 , srv915425050:84 , srv946485209:84 , srv955388772:84 , srv965088415:84 , srv98609981:84 , srv98616297:84 , srv995619677:84 , srv296499600:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-17 20:53:40,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:40,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181358362=3, srv670879095=73, srv1550553316=18, srv562554696=63, srv1442694669=12, srv1678090744=25, srv2091932160=48, srv1257032753=5, srv679693484=75, srv617799203=67, srv298008411=57, srv668426153=72, srv1313699472=8, srv1193360237=4, srv142456192=11, srv1269778082=7, srv87974878=88, srv914657124=92, srv1379050822=10, srv21354799=54, srv1785966590=30, srv1496817440=15, srv915425050=93, srv1725590265=27, srv1612810700=21, srv965088415=96, srv1644333093=22, srv1757244962=28, srv679204579=74, srv1670116464=23, srv767828861=81, srv869997298=85, srv714015773=78, srv1257352809=6, srv1950069499=43, srv25290698=55, srv995619677=99, srv423540818=60, srv1569570377=19, srv751541073=80, srv1876191076=37, srv878700085=87, srv401006438=59, srv1821803894=34, srv833569513=83, srv574404471=65, srv1947750038=41, srv596472655=66, srv1949320994=42, srv783205531=82, srv2058648266=46, srv726992922=79, srv1104562388=2, srv2000347201=44, srv1480441065=14, srv98616297=98, srv651929968=70, srv3349383=58, srv1797866175=31, srv2086927628=47, srv908366295=91, srv211541025=51, srv2103564377=50, srv1935774375=38, srv1047413213=1, srv1512569711=16, srv1870036789=36, srv1450110954=13, srv1821370155=33, srv2117746073=52, srv2119116659=53, srv632067014=68, srv1597347682=20, srv1941182402=39, srv1352302196=9, srv1695340801=26, srv867556935=84, srv1677955006=24, srv684108754=76, srv569780=64, srv98609981=97, srv1848095134=35, srv296499600=56, srv66151263=71, srv1945110437=40, srv900800212=90, srv1009167469=0, srv493251843=62, srv632696262=69, srv707537188=77, srv2095749286=49, srv1765104397=29, srv2004452166=45, srv1549064741=17, srv478785675=61, srv181058790=32, srv955388772=95, srv878673235=86, srv946485209=94, srv8992972=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:40,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:40,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:40,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:40,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:40,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:40,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:40,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:40,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-17 20:53:40,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-17 20:53:40,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-17 20:53:40,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-17 20:53:40,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-17 20:53:40,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-17 20:53:40,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-17 20:53:40,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-17 20:53:40,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-17 20:53:40,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-17 20:53:40,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-17 20:53:40,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-17 20:53:40,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-17 20:53:40,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-17 20:53:40,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-17 20:53:40,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-17 20:53:40,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-17 20:53:40,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-17 20:53:40,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-17 20:53:40,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-17 20:53:40,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-17 20:53:40,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-17 20:53:40,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-17 20:53:40,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-17 20:53:40,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-17 20:53:40,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-17 20:53:40,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-17 20:53:40,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-17 20:53:40,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-17 20:53:40,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-17 20:53:40,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-17 20:53:40,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-17 20:53:40,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-17 20:53:40,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-17 20:53:40,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-17 20:53:40,603 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:40,604 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-17 20:53:40,605 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-17 20:53:40,989 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 571 ms to try 5000 different iterations. Found a solution that moves 4708 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18113804323263277. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04008999727216198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05365433699886597, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5231111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04008999727216198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23933333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:41,036 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1935774375:81 , srv1797866175:82 , srv181058790:82 , srv2086927628:82 , srv562554696:82 , srv574404471:82 , srv707537188:82 , srv878700085:82 , srv1677955006:83 , srv1941182402:83 , srv1949320994:83 , srv2058648266:83 , srv3349383:83 , srv596472655:83 , srv632067014:83 , srv66151263:83 , srv726992922:83 , srv869997298:83 , srv946485209:83 , srv1009167469:84 , srv1765104397:84 , srv1876191076:84 , srv21354799:84 , srv25290698:84 , srv679693484:84 , srv767828861:84 , srv900800212:84 , srv98609981:84 , srv1442694669:85 , srv1480441065:85 , srv1569570377:85 , srv2004452166:85 , srv2103564377:85 , srv651929968:85 , srv679204579:85 , srv783205531:85 , srv833569513:85 , srv8992972:85 , srv98616297:85 , srv1104562388:86 , srv142456192:86 , srv1670116464:86 , srv1757244962:86 , srv1848095134:86 , srv1870036789:86 , srv2000347201:86 , srv478785675:86 , srv617799203:86 , srv632696262:86 , srv867556935:86 , srv908366295:86 , srv955388772:86 , srv1047413213:87 , srv1269778082:87 , srv1450110954:87 , srv1496817440:87 , srv1550553316:87 , srv1695340801:87 , srv1785966590:87 , srv423540818:87 , srv569780:87 , srv668426153:87 , srv670879095:87 , srv87974878:87 , srv914657124:87 , srv915425050:87 , srv995619677:87 , srv1181358362:88 , srv1193360237:88 , srv1257352809:88 , srv1379050822:88 , srv1549064741:88 , srv1678090744:88 , srv1821370155:88 , srv1950069499:88 , srv211541025:88 , srv684108754:88 , srv751541073:88 , srv878673235:88 , srv1352302196:89 , srv1512569711:89 , srv1597347682:89 , srv1644333093:89 , srv1821803894:89 , srv1945110437:89 , srv1947750038:89 , srv1257032753:90 , srv1313699472:90 , srv2095749286:90 , srv493251843:90 , srv2117746073:91 , srv714015773:91 , srv965088415:91 , srv2119116659:92 , srv298008411:92 , srv1612810700:93 , srv401006438:93 , srv2091932160:94 , srv1725590265:96 , srv296499600:446 } 2023-05-17 20:53:41,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:41,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181358362=3, srv670879095=73, srv1550553316=18, srv562554696=63, srv1442694669=12, srv1678090744=25, srv2091932160=48, srv1257032753=5, srv679693484=75, srv617799203=67, srv298008411=57, srv668426153=72, srv1313699472=8, srv1193360237=4, srv142456192=11, srv1269778082=7, srv87974878=88, srv914657124=92, srv1379050822=10, srv21354799=54, srv1785966590=30, srv1496817440=15, srv915425050=93, srv1725590265=27, srv1612810700=21, srv965088415=96, srv1644333093=22, srv1757244962=28, srv679204579=74, srv1670116464=23, srv767828861=81, srv869997298=85, srv714015773=78, srv1257352809=6, srv1950069499=43, srv25290698=55, srv995619677=99, srv423540818=60, srv1569570377=19, srv751541073=80, srv1876191076=37, srv878700085=87, srv401006438=59, srv1821803894=34, srv833569513=83, srv574404471=65, srv1947750038=41, srv596472655=66, srv1949320994=42, srv783205531=82, srv2058648266=46, srv726992922=79, srv1104562388=2, srv2000347201=44, srv1480441065=14, srv98616297=98, srv651929968=70, srv3349383=58, srv1797866175=31, srv2086927628=47, srv908366295=91, srv211541025=51, srv2103564377=50, srv1935774375=38, srv1047413213=1, srv1512569711=16, srv1870036789=36, srv1450110954=13, srv1821370155=33, srv2117746073=52, srv2119116659=53, srv632067014=68, srv1597347682=20, srv1941182402=39, srv1352302196=9, srv1695340801=26, srv867556935=84, srv1677955006=24, srv684108754=76, srv569780=64, srv98609981=97, srv1848095134=35, srv296499600=56, srv66151263=71, srv1945110437=40, srv900800212=90, srv1009167469=0, srv493251843=62, srv632696262=69, srv707537188=77, srv2095749286=49, srv1765104397=29, srv2004452166=45, srv1549064741=17, srv478785675=61, srv181058790=32, srv955388772=95, srv878673235=86, srv946485209=94, srv8992972=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:41,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:41,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:41,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:41,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-17 20:53:41,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-17 20:53:41,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-17 20:53:41,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-17 20:53:41,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-17 20:53:41,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-17 20:53:41,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-17 20:53:41,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-17 20:53:41,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-17 20:53:41,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-17 20:53:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-17 20:53:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-17 20:53:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-17 20:53:41,253 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04008999727216198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05365433699886597, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04008999727216198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23933333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:41,254 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-17 20:53:41,254 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18110507711203375, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04008999727216198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05365433699886597, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04008999727216198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23933333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-17 20:53:41,610 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 485 ms to try 5000 different iterations. Found a solution that moves 2616 regions; Going from a computed imbalance of 0.18110507711203375 to a new imbalance of 0.014777054479904132. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02161920324321482); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030579880125049568, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2906666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02161920324321482); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:41,620 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1725590265:75 , srv2091932160:77 , srv401006438:77 , srv1612810700:80 , srv714015773:80 , srv1313699472:81 , srv1821803894:81 , srv298008411:81 , srv1379050822:82 , srv2117746073:82 , srv2119116659:82 , srv965088415:82 , srv1269778082:83 , srv1352302196:83 , srv1512569711:83 , srv1549064741:83 , srv1597347682:83 , srv1678090744:83 , srv1848095134:83 , srv1945110437:83 , srv2095749286:83 , srv493251843:83 , srv1193360237:84 , srv1257032753:84 , srv1550553316:84 , srv1695340801:84 , srv1821370155:84 , srv2000347201:84 , srv211541025:84 , srv423540818:84 , srv478785675:84 , srv668426153:84 , srv684108754:84 , srv915425050:84 , srv1104562388:85 , srv1181358362:85 , srv1757244962:85 , srv1876191076:85 , srv25290698:85 , srv569780:85 , srv617799203:85 , srv867556935:85 , srv87974878:85 , srv900800212:85 , srv995619677:85 , srv1257352809:86 , srv1450110954:86 , srv1480441065:86 , srv1496817440:86 , srv1785966590:86 , srv1870036789:86 , srv1947750038:86 , srv1950069499:86 , srv2004452166:86 , srv2103564377:86 , srv751541073:86 , srv914657124:86 , srv955388772:86 , srv1047413213:87 , srv1442694669:87 , srv1569570377:87 , srv1670116464:87 , srv1941182402:87 , srv21354799:87 , srv651929968:87 , srv670879095:87 , srv679204579:87 , srv679693484:87 , srv767828861:87 , srv833569513:87 , srv878673235:87 , srv908366295:87 , srv98616297:87 , srv1644333093:88 , srv1677955006:88 , srv1765104397:88 , srv2058648266:88 , srv3349383:88 , srv596472655:88 , srv632067014:88 , srv632696262:88 , srv783205531:88 , srv878700085:88 , srv98609981:88 , srv142456192:89 , srv181058790:89 , srv1949320994:89 , srv562554696:89 , srv8992972:89 , srv1009167469:90 , srv2086927628:90 , srv726992922:90 , srv869997298:90 , srv1797866175:91 , srv1935774375:91 , srv574404471:91 , srv66151263:91 , srv946485209:92 , srv707537188:95 , srv296499600:520 } 2023-05-17 20:53:41,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:41,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181358362=3, srv670879095=73, srv1550553316=18, srv562554696=63, srv1442694669=12, srv1678090744=25, srv2091932160=48, srv1257032753=5, srv679693484=75, srv617799203=67, srv298008411=57, srv668426153=72, srv1313699472=8, srv1193360237=4, srv142456192=11, srv1269778082=7, srv87974878=88, srv914657124=92, srv1379050822=10, srv21354799=54, srv1785966590=30, srv1496817440=15, srv915425050=93, srv1725590265=27, srv1612810700=21, srv965088415=96, srv1644333093=22, srv1757244962=28, srv679204579=74, srv1670116464=23, srv767828861=81, srv869997298=85, srv714015773=78, srv1257352809=6, srv1950069499=43, srv25290698=55, srv995619677=99, srv423540818=60, srv1569570377=19, srv751541073=80, srv1876191076=37, srv878700085=87, srv401006438=59, srv1821803894=34, srv833569513=83, srv574404471=65, srv1947750038=41, srv596472655=66, srv1949320994=42, srv783205531=82, srv2058648266=46, srv726992922=79, srv1104562388=2, srv2000347201=44, srv1480441065=14, srv98616297=98, srv651929968=70, srv3349383=58, srv1797866175=31, srv2086927628=47, srv908366295=91, srv211541025=51, srv2103564377=50, srv1935774375=38, srv1047413213=1, srv1512569711=16, srv1870036789=36, srv1450110954=13, srv1821370155=33, srv2117746073=52, srv2119116659=53, srv632067014=68, srv1597347682=20, srv1941182402=39, srv1352302196=9, srv1695340801=26, srv867556935=84, srv1677955006=24, srv684108754=76, srv569780=64, srv98609981=97, srv1848095134=35, srv296499600=56, srv66151263=71, srv1945110437=40, srv900800212=90, srv1009167469=0, srv493251843=62, srv632696262=69, srv707537188=77, srv2095749286=49, srv1765104397=29, srv2004452166=45, srv1549064741=17, srv478785675=61, srv181058790=32, srv955388772=95, srv878673235=86, srv946485209=94, srv8992972=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:41,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:41,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:41,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-17 20:53:41,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-17 20:53:41,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-17 20:53:41,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-17 20:53:41,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-17 20:53:41,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-17 20:53:41,783 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02161920324321482); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030579880125049568, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02161920324321482); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:41,783 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-17 20:53:41,784 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014758736856393714, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02161920324321482); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030579880125049568, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02161920324321482); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-17 20:53:42,108 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 911 regions; Going from a computed imbalance of 0.014758736856393714 to a new imbalance of 6.668742098326946E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072198476628214546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007339758434175737); MoveCostFunction : (multiplier=7.0, imbalance=0.10122222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072198476628214546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:42,110 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1644333093:81 , srv707537188:81 , srv1009167469:82 , srv878673235:82 , srv1257352809:83 , srv142456192:83 , srv1947750038:83 , srv66151263:83 , srv908366295:83 , srv946485209:83 , srv1047413213:84 , srv1193360237:84 , srv1257032753:84 , srv1450110954:84 , srv1496817440:84 , srv1935774375:84 , srv1945110437:84 , srv1950069499:84 , srv2095749286:84 , srv211541025:84 , srv2117746073:84 , srv2119116659:84 , srv298008411:84 , srv632696262:84 , srv670879095:84 , srv684108754:84 , srv751541073:84 , srv869997298:84 , srv8992972:84 , srv965088415:84 , srv1442694669:85 , srv1569570377:85 , srv1597347682:85 , srv1612810700:85 , srv1670116464:85 , srv1765104397:85 , srv1785966590:85 , srv1797866175:85 , srv1821370155:85 , srv1870036789:85 , srv1949320994:85 , srv493251843:85 , srv569780:85 , srv574404471:85 , srv651929968:85 , srv679204579:85 , srv726992922:85 , srv87974878:85 , srv914657124:85 , srv955388772:85 , srv98609981:85 , srv98616297:85 , srv1104562388:86 , srv1181358362:86 , srv1313699472:86 , srv1352302196:86 , srv1512569711:86 , srv1677955006:86 , srv1678090744:86 , srv1757244962:86 , srv1941182402:86 , srv2004452166:86 , srv2058648266:86 , srv2086927628:86 , srv2091932160:86 , srv21354799:86 , srv3349383:86 , srv596472655:86 , srv617799203:86 , srv632067014:86 , srv668426153:86 , srv679693484:86 , srv714015773:86 , srv767828861:86 , srv783205531:86 , srv833569513:86 , srv915425050:86 , srv995619677:86 , srv1379050822:87 , srv1480441065:87 , srv1549064741:87 , srv1550553316:87 , srv1695340801:87 , srv1725590265:87 , srv181058790:87 , srv2103564377:87 , srv401006438:87 , srv423540818:87 , srv478785675:87 , srv562554696:87 , srv867556935:87 , srv878700085:87 , srv1269778082:88 , srv1821803894:88 , srv1848095134:88 , srv1876191076:88 , srv2000347201:88 , srv25290698:88 , srv900800212:88 , srv296499600:556 } 2023-05-17 20:53:42,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:42,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181358362=3, srv670879095=73, srv1550553316=18, srv562554696=63, srv1442694669=12, srv1678090744=25, srv2091932160=48, srv1257032753=5, srv679693484=75, srv617799203=67, srv298008411=57, srv668426153=72, srv1313699472=8, srv1193360237=4, srv142456192=11, srv1269778082=7, srv87974878=88, srv914657124=92, srv1379050822=10, srv21354799=54, srv1785966590=30, srv1496817440=15, srv915425050=93, srv1725590265=27, srv1612810700=21, srv965088415=96, srv1644333093=22, srv1757244962=28, srv679204579=74, srv1670116464=23, srv767828861=81, srv869997298=85, srv714015773=78, srv1257352809=6, srv1950069499=43, srv25290698=55, srv995619677=99, srv423540818=60, srv1569570377=19, srv751541073=80, srv1876191076=37, srv878700085=87, srv401006438=59, srv1821803894=34, srv833569513=83, srv574404471=65, srv1947750038=41, srv596472655=66, srv1949320994=42, srv783205531=82, srv2058648266=46, srv726992922=79, srv1104562388=2, srv2000347201=44, srv1480441065=14, srv98616297=98, srv651929968=70, srv3349383=58, srv1797866175=31, srv2086927628=47, srv908366295=91, srv211541025=51, srv2103564377=50, srv1935774375=38, srv1047413213=1, srv1512569711=16, srv1870036789=36, srv1450110954=13, srv1821370155=33, srv2117746073=52, srv2119116659=53, srv632067014=68, srv1597347682=20, srv1941182402=39, srv1352302196=9, srv1695340801=26, srv867556935=84, srv1677955006=24, srv684108754=76, srv569780=64, srv98609981=97, srv1848095134=35, srv296499600=56, srv66151263=71, srv1945110437=40, srv900800212=90, srv1009167469=0, srv493251843=62, srv632696262=69, srv707537188=77, srv2095749286=49, srv1765104397=29, srv2004452166=45, srv1549064741=17, srv478785675=61, srv181058790=32, srv955388772=95, srv878673235=86, srv946485209=94, srv8992972=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:42,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:42,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:42,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:42,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-17 20:53:42,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-17 20:53:42,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-17 20:53:42,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-17 20:53:42,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-17 20:53:42,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-17 20:53:42,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-17 20:53:42,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-17 20:53:42,250 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072198476628214546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007339758434175737); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072198476628214546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:42,250 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-17 20:53:42,250 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.604952514924843E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072198476628214546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007339758434175737); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072198476628214546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-17 20:53:42,547 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 394 ms to try 5000 different iterations. Found a solution that moves 261 regions; Going from a computed imbalance of 6.604952514924843E-4 to a new imbalance of 4.407881404046928E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.116465361813114E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.116465361813114E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:42,548 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181358362:83 , srv181058790:83 , srv783205531:83 , srv867556935:83 , srv914657124:83 , srv946485209:83 , srv1047413213:84 , srv1257032753:84 , srv1269778082:84 , srv1313699472:84 , srv1352302196:84 , srv142456192:84 , srv1442694669:84 , srv1480441065:84 , srv1512569711:84 , srv1550553316:84 , srv1612810700:84 , srv1670116464:84 , srv1695340801:84 , srv1757244962:84 , srv1785966590:84 , srv1797866175:84 , srv1821803894:84 , srv1870036789:84 , srv1876191076:84 , srv1947750038:84 , srv1950069499:84 , srv2000347201:84 , srv2086927628:84 , srv2103564377:84 , srv2117746073:84 , srv2119116659:84 , srv423540818:84 , srv562554696:84 , srv574404471:84 , srv632696262:84 , srv668426153:84 , srv670879095:84 , srv726992922:84 , srv833569513:84 , srv869997298:84 , srv8992972:84 , srv995619677:84 , srv1009167469:85 , srv1104562388:85 , srv1257352809:85 , srv1379050822:85 , srv1450110954:85 , srv1496817440:85 , srv1549064741:85 , srv1569570377:85 , srv1597347682:85 , srv1644333093:85 , srv1677955006:85 , srv1725590265:85 , srv1765104397:85 , srv1821370155:85 , srv1935774375:85 , srv2004452166:85 , srv2058648266:85 , srv2091932160:85 , srv21354799:85 , srv25290698:85 , srv3349383:85 , srv401006438:85 , srv493251843:85 , srv569780:85 , srv617799203:85 , srv632067014:85 , srv651929968:85 , srv66151263:85 , srv679204579:85 , srv684108754:85 , srv707537188:85 , srv714015773:85 , srv751541073:85 , srv767828861:85 , srv878673235:85 , srv878700085:85 , srv87974878:85 , srv900800212:85 , srv915425050:85 , srv955388772:85 , srv965088415:85 , srv98609981:85 , srv98616297:85 , srv1193360237:86 , srv1678090744:86 , srv1848095134:86 , srv1945110437:86 , srv1949320994:86 , srv2095749286:86 , srv211541025:86 , srv298008411:86 , srv478785675:86 , srv596472655:86 , srv679693484:86 , srv908366295:86 , srv1941182402:87 , srv296499600:620 } 2023-05-17 20:53:42,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:42,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181358362=3, srv670879095=73, srv1550553316=18, srv562554696=63, srv1442694669=12, srv1678090744=25, srv2091932160=48, srv1257032753=5, srv679693484=75, srv617799203=67, srv298008411=57, srv668426153=72, srv1313699472=8, srv1193360237=4, srv142456192=11, srv1269778082=7, srv87974878=88, srv914657124=92, srv1379050822=10, srv21354799=54, srv1785966590=30, srv1496817440=15, srv915425050=93, srv1725590265=27, srv1612810700=21, srv965088415=96, srv1644333093=22, srv1757244962=28, srv679204579=74, srv1670116464=23, srv767828861=81, srv869997298=85, srv714015773=78, srv1257352809=6, srv1950069499=43, srv25290698=55, srv995619677=99, srv423540818=60, srv1569570377=19, srv751541073=80, srv1876191076=37, srv878700085=87, srv401006438=59, srv1821803894=34, srv833569513=83, srv574404471=65, srv1947750038=41, srv596472655=66, srv1949320994=42, srv783205531=82, srv2058648266=46, srv726992922=79, srv1104562388=2, srv2000347201=44, srv1480441065=14, srv98616297=98, srv651929968=70, srv3349383=58, srv1797866175=31, srv2086927628=47, srv908366295=91, srv211541025=51, srv2103564377=50, srv1935774375=38, srv1047413213=1, srv1512569711=16, srv1870036789=36, srv1450110954=13, srv1821370155=33, srv2117746073=52, srv2119116659=53, srv632067014=68, srv1597347682=20, srv1941182402=39, srv1352302196=9, srv1695340801=26, srv867556935=84, srv1677955006=24, srv684108754=76, srv569780=64, srv98609981=97, srv1848095134=35, srv296499600=56, srv66151263=71, srv1945110437=40, srv900800212=90, srv1009167469=0, srv493251843=62, srv632696262=69, srv707537188=77, srv2095749286=49, srv1765104397=29, srv2004452166=45, srv1549064741=17, srv478785675=61, srv181058790=32, srv955388772=95, srv878673235=86, srv946485209=94, srv8992972=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:42,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:42,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:42,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:42,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:42,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-17 20:53:42,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-17 20:53:42,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-17 20:53:42,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-17 20:53:42,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-17 20:53:42,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-17 20:53:42,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-17 20:53:42,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-17 20:53:42,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-17 20:53:42,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-17 20:53:42,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-17 20:53:42,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-17 20:53:42,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-17 20:53:42,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-17 20:53:42,677 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.116465361813114E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.116465361813114E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:42,677 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-17 20:53:42,678 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.2251252979223486E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.116465361813114E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.116465361813114E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-17 20:53:42,938 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 356 ms to try 5000 different iterations. Found a solution that moves 37 regions; Going from a computed imbalance of 4.2251252979223486E-5 to a new imbalance of 7.948501337768768E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:42,938 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1549064741:83 , srv1644333093:83 , srv1678090744:83 , srv1725590265:83 , srv1848095134:83 , srv1941182402:83 , srv1949320994:83 , srv2095749286:83 , srv298008411:83 , srv478785675:83 , srv493251843:83 , srv596472655:83 , srv679693484:83 , srv707537188:83 , srv751541073:83 , srv1047413213:84 , srv1104562388:84 , srv1181358362:84 , srv1193360237:84 , srv1257032753:84 , srv1257352809:84 , srv1269778082:84 , srv1352302196:84 , srv1379050822:84 , srv142456192:84 , srv1450110954:84 , srv1480441065:84 , srv1496817440:84 , srv1512569711:84 , srv1550553316:84 , srv1569570377:84 , srv1597347682:84 , srv1612810700:84 , srv1670116464:84 , srv1677955006:84 , srv1695340801:84 , srv1765104397:84 , srv1785966590:84 , srv1797866175:84 , srv1821370155:84 , srv1821803894:84 , srv1945110437:84 , srv1947750038:84 , srv1950069499:84 , srv2000347201:84 , srv2004452166:84 , srv2058648266:84 , srv2086927628:84 , srv2091932160:84 , srv2103564377:84 , srv211541025:84 , srv2119116659:84 , srv21354799:84 , srv25290698:84 , srv3349383:84 , srv401006438:84 , srv423540818:84 , srv562554696:84 , srv569780:84 , srv574404471:84 , srv617799203:84 , srv632067014:84 , srv632696262:84 , srv651929968:84 , srv66151263:84 , srv670879095:84 , srv679204579:84 , srv714015773:84 , srv726992922:84 , srv767828861:84 , srv783205531:84 , srv833569513:84 , srv878673235:84 , srv878700085:84 , srv87974878:84 , srv8992972:84 , srv900800212:84 , srv908366295:84 , srv915425050:84 , srv955388772:84 , srv965088415:84 , srv98609981:84 , srv98616297:84 , srv995619677:84 , srv1009167469:85 , srv1313699472:85 , srv1442694669:85 , srv1757244962:85 , srv181058790:85 , srv1870036789:85 , srv1876191076:85 , srv1935774375:85 , srv2117746073:85 , srv668426153:85 , srv684108754:85 , srv867556935:85 , srv869997298:85 , srv914657124:85 , srv946485209:85 , srv296499600:684 } 2023-05-17 20:53:42,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:42,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181358362=3, srv670879095=73, srv1550553316=18, srv562554696=63, srv1442694669=12, srv1678090744=25, srv2091932160=48, srv1257032753=5, srv679693484=75, srv617799203=67, srv298008411=57, srv668426153=72, srv1313699472=8, srv1193360237=4, srv142456192=11, srv1269778082=7, srv87974878=88, srv914657124=92, srv1379050822=10, srv21354799=54, srv1785966590=30, srv1496817440=15, srv915425050=93, srv1725590265=27, srv1612810700=21, srv965088415=96, srv1644333093=22, srv1757244962=28, srv679204579=74, srv1670116464=23, srv767828861=81, srv869997298=85, srv714015773=78, srv1257352809=6, srv1950069499=43, srv25290698=55, srv995619677=99, srv423540818=60, srv1569570377=19, srv751541073=80, srv1876191076=37, srv878700085=87, srv401006438=59, srv1821803894=34, srv833569513=83, srv574404471=65, srv1947750038=41, srv596472655=66, srv1949320994=42, srv783205531=82, srv2058648266=46, srv726992922=79, srv1104562388=2, srv2000347201=44, srv1480441065=14, srv98616297=98, srv651929968=70, srv3349383=58, srv1797866175=31, srv2086927628=47, srv908366295=91, srv211541025=51, srv2103564377=50, srv1935774375=38, srv1047413213=1, srv1512569711=16, srv1870036789=36, srv1450110954=13, srv1821370155=33, srv2117746073=52, srv2119116659=53, srv632067014=68, srv1597347682=20, srv1941182402=39, srv1352302196=9, srv1695340801=26, srv867556935=84, srv1677955006=24, srv684108754=76, srv569780=64, srv98609981=97, srv1848095134=35, srv296499600=56, srv66151263=71, srv1945110437=40, srv900800212=90, srv1009167469=0, srv493251843=62, srv632696262=69, srv707537188=77, srv2095749286=49, srv1765104397=29, srv2004452166=45, srv1549064741=17, srv478785675=61, srv181058790=32, srv955388772=95, srv878673235=86, srv946485209=94, srv8992972=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:42,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:42,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-17 20:53:42,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-17 20:53:42,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-17 20:53:42,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-17 20:53:42,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-17 20:53:42,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-17 20:53:42,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-17 20:53:42,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-17 20:53:42,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-17 20:53:42,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-17 20:53:43,063 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:43,063 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1549064741:83 , srv1644333093:83 , srv1678090744:83 , srv1725590265:83 , srv1848095134:83 , srv1941182402:83 , srv1949320994:83 , srv2095749286:83 , srv298008411:83 , srv478785675:83 , srv493251843:83 , srv596472655:83 , srv679693484:83 , srv707537188:83 , srv751541073:83 , srv1047413213:84 , srv1104562388:84 , srv1181358362:84 , srv1193360237:84 , srv1257032753:84 , srv1257352809:84 , srv1269778082:84 , srv1352302196:84 , srv1379050822:84 , srv142456192:84 , srv1450110954:84 , srv1480441065:84 , srv1496817440:84 , srv1512569711:84 , srv1550553316:84 , srv1569570377:84 , srv1597347682:84 , srv1612810700:84 , srv1670116464:84 , srv1677955006:84 , srv1695340801:84 , srv1765104397:84 , srv1785966590:84 , srv1797866175:84 , srv1821370155:84 , srv1821803894:84 , srv1945110437:84 , srv1947750038:84 , srv1950069499:84 , srv2000347201:84 , srv2004452166:84 , srv2058648266:84 , srv2086927628:84 , srv2091932160:84 , srv2103564377:84 , srv211541025:84 , srv2119116659:84 , srv21354799:84 , srv25290698:84 , srv3349383:84 , srv401006438:84 , srv423540818:84 , srv562554696:84 , srv569780:84 , srv574404471:84 , srv617799203:84 , srv632067014:84 , srv632696262:84 , srv651929968:84 , srv66151263:84 , srv670879095:84 , srv679204579:84 , srv714015773:84 , srv726992922:84 , srv767828861:84 , srv783205531:84 , srv833569513:84 , srv878673235:84 , srv878700085:84 , srv87974878:84 , srv8992972:84 , srv900800212:84 , srv908366295:84 , srv915425050:84 , srv955388772:84 , srv965088415:84 , srv98609981:84 , srv98616297:84 , srv995619677:84 , srv1009167469:85 , srv1313699472:85 , srv1442694669:85 , srv1757244962:85 , srv181058790:85 , srv1870036789:85 , srv1876191076:85 , srv1935774375:85 , srv2117746073:85 , srv668426153:85 , srv684108754:85 , srv867556935:85 , srv869997298:85 , srv914657124:85 , srv946485209:85 , srv296499600:684 } 2023-05-17 20:53:43,090 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=317 (was 317), ProcessCount=174 (was 174), AvailableMemoryMB=3781 (was 4287) 2023-05-17 20:53:43,100 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=174, AvailableMemoryMB=3780 2023-05-17 20:53:43,101 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-17 20:53:43,101 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-17 20:53:43,113 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026659277:84 , srv1046689550:84 , srv1115331240:84 , srv1165357631:84 , srv1192688679:84 , srv1261962125:84 , srv1296431933:84 , srv137644953:84 , srv1416035301:84 , srv1478218135:84 , srv1662638917:84 , srv1717405768:84 , srv1763448920:84 , srv1845380602:84 , srv1865679161:84 , srv200484460:84 , srv2088608913:84 , srv272989840:84 , srv477610912:84 , srv493447342:84 , srv504311945:84 , srv527522364:84 , srv591981538:84 , srv594778828:84 , srv611711992:84 , srv650183428:84 , srv709356897:84 , srv817365909:84 , srv970214174:84 , srv116686919:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-17 20:53:43,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:53:43,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2088608913=17, srv1261962125=6, srv1115331240=2, srv477610912=19, srv493447342=20, srv1165357631=3, srv650183428=26, srv1717405768=12, srv1845380602=14, srv591981538=23, srv709356897=27, srv594778828=24, srv1662638917=11, srv272989840=18, srv137644953=8, srv527522364=22, srv504311945=21, srv1296431933=7, srv1478218135=10, srv611711992=25, srv1026659277=0, srv1192688679=5, srv1865679161=15, srv1763448920=13, srv1416035301=9, srv200484460=16, srv970214174=29, srv817365909=28, srv1046689550=1, srv116686919=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:53:43,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:53:43,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:53:43,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-17 20:53:43,156 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1079940008605849, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:53:43,156 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912533328834663, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1079940008605849, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-17 20:55:43,134 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5696626 different iterations. Found a solution that moves 1870 regions; Going from a computed imbalance of 0.9912533328834663 to a new imbalance of 4.3646732880327594E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6925925925925925, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:55:43,141 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1026659277:90 , srv1046689550:90 , srv1115331240:90 , srv1165357631:90 , srv116686919:90 , srv1192688679:90 , srv1261962125:90 , srv1296431933:90 , srv137644953:90 , srv1416035301:90 , srv1478218135:90 , srv1662638917:90 , srv1717405768:90 , srv1763448920:90 , srv1845380602:90 , srv1865679161:90 , srv200484460:90 , srv2088608913:90 , srv272989840:90 , srv477610912:90 , srv493447342:90 , srv504311945:90 , srv527522364:90 , srv591981538:90 , srv594778828:90 , srv611711992:90 , srv650183428:90 , srv709356897:90 , srv817365909:90 , srv970214174:90 } 2023-05-17 20:55:43,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-17 20:55:43,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2088608913=17, srv1261962125=6, srv1115331240=2, srv477610912=19, srv493447342=20, srv1165357631=3, srv650183428=26, srv1717405768=12, srv1845380602=14, srv591981538=23, srv709356897=27, srv594778828=24, srv1662638917=11, srv272989840=18, srv137644953=8, srv527522364=22, srv504311945=21, srv1296431933=7, srv1478218135=10, srv611711992=25, srv1026659277=0, srv1192688679=5, srv1865679161=15, srv1763448920=13, srv1416035301=9, srv200484460=16, srv970214174=29, srv817365909=28, srv1046689550=1, srv116686919=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-17 20:55:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-17 20:55:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-17 20:55:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-17 20:55:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-17 20:55:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-17 20:55:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-17 20:55:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-17 20:55:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-17 20:55:43,166 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-17 20:55:43,166 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1026659277:90 , srv1046689550:90 , srv1115331240:90 , srv1165357631:90 , srv116686919:90 , srv1192688679:90 , srv1261962125:90 , srv1296431933:90 , srv137644953:90 , srv1416035301:90 , srv1478218135:90 , srv1662638917:90 , srv1717405768:90 , srv1763448920:90 , srv1845380602:90 , srv1865679161:90 , srv200484460:90 , srv2088608913:90 , srv272989840:90 , srv477610912:90 , srv493447342:90 , srv504311945:90 , srv527522364:90 , srv591981538:90 , srv594778828:90 , srv611711992:90 , srv650183428:90 , srv709356897:90 , srv817365909:90 , srv970214174:90 } 2023-05-17 20:55:43,183 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=233 (was 317), ProcessCount=170 (was 174), AvailableMemoryMB=3609 (was 3780)