2023-09-23 20:52:44,850 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-23 20:52:45,138 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-23 20:52:45,231 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-23 20:52:45,286 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 20:52:45,292 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-23 20:52:45,304 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=321, ProcessCount=175, AvailableMemoryMB=22595 2023-09-23 20:52:45,306 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 20:52:45,307 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-23 20:52:45,459 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002937901:84 , srv100676573:84 , srv1012263267:84 , srv1040909541:84 , srv1079923235:84 , srv1093883575:84 , srv1110912192:84 , srv1147068467:84 , srv1162560508:84 , srv1188436843:84 , srv1200189448:84 , srv1204379949:84 , srv1204594708:84 , srv121851787:84 , srv1227331566:84 , srv1285576371:84 , srv1288883104:84 , srv1318976506:84 , srv1323186939:84 , srv1335113165:84 , srv1341430296:84 , srv1344233399:84 , srv1360142925:84 , srv1370846652:84 , srv1378064232:84 , srv1427749919:84 , srv1428875004:84 , srv143142669:84 , srv1436768975:84 , srv1448644977:84 , srv145698554:84 , srv1481010184:84 , srv1489560683:84 , srv1504108283:84 , srv1569104242:84 , srv1592952573:84 , srv1624814483:84 , srv1625352951:84 , srv1680782224:84 , srv1683124142:84 , srv1716772794:84 , srv1718143094:84 , srv1733294273:84 , srv1760445923:84 , srv1806400400:84 , srv1820827041:84 , srv1822091113:84 , srv1826127928:84 , srv1833041721:84 , srv1907616528:84 , srv1936499168:84 , srv1989121199:84 , srv1997687412:84 , srv2002048809:84 , srv2007167689:84 , srv201962836:84 , srv2025469200:84 , srv2032324913:84 , srv2039095464:84 , srv2077968598:84 , srv2114887160:84 , srv2130272343:84 , srv2136593238:84 , srv22448346:84 , srv234544888:84 , srv243499026:84 , srv268709865:84 , srv297464682:84 , srv316259073:84 , srv316643571:84 , srv323102590:84 , srv33329397:84 , srv343866517:84 , srv373435707:84 , srv380679175:84 , srv385475962:84 , srv404881017:84 , srv416582570:84 , srv422277851:84 , srv446364696:84 , srv45293674:84 , srv645134418:84 , srv656565098:84 , srv664085787:84 , srv665925944:84 , srv672531609:84 , srv704063843:84 , srv718126802:84 , srv813950210:84 , srv817979368:84 , srv833011667:84 , srv846382561:84 , srv887096966:84 , srv891151537:84 , srv89243407:84 , srv911678090:84 , srv916237271:84 , srv956417765:84 , srv969278174:84 , srv528272546:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-23 20:52:45,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:52:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2114887160=60, srv1683124142=39, srv1716772794=40, srv1227331566=14, srv121851787=13, srv45293674=80, srv1288883104=16, srv2130272343=61, srv1378064232=24, srv1822091113=46, srv145698554=30, srv1427749919=25, srv316643571=69, srv1428875004=26, srv1200189448=10, srv343866517=72, srv373435707=73, srv1989121199=51, srv817979368=90, srv1718143094=41, srv1733294273=42, srv1040909541=3, srv813950210=89, srv1341430296=20, srv1110912192=6, srv1569104242=34, srv833011667=91, srv1448644977=29, srv1360142925=22, srv2039095464=58, srv1079923235=4, srv316259073=68, srv385475962=75, srv1833041721=48, srv201962836=55, srv1323186939=18, srv1489560683=32, srv1680782224=38, srv89243407=95, srv1625352951=37, srv665925944=85, srv1936499168=50, srv2077968598=59, srv645134418=82, srv297464682=67, srv33329397=71, srv100676573=1, srv887096966=93, srv1760445923=43, srv1162560508=8, srv656565098=83, srv323102590=70, srv268709865=66, srv1504108283=33, srv1826127928=47, srv1481010184=31, srv243499026=65, srv1318976506=17, srv422277851=78, srv1204379949=11, srv2032324913=57, srv2002048809=53, srv916237271=97, srv380679175=74, srv2136593238=62, srv416582570=77, srv891151537=94, srv1093883575=5, srv404881017=76, srv1204594708=12, srv234544888=64, srv846382561=92, srv911678090=96, srv1285576371=15, srv143142669=27, srv1436768975=28, srv1806400400=44, srv718126802=88, srv664085787=84, srv2025469200=56, srv1592952573=35, srv1820827041=45, srv22448346=63, srv1344233399=21, srv1188436843=9, srv672531609=86, srv528272546=81, srv1012263267=2, srv1147068467=7, srv1997687412=52, srv704063843=87, srv1370846652=23, srv1907616528=49, srv446364696=79, srv1624814483=36, srv956417765=98, srv2007167689=54, srv969278174=99, srv1002937901=0, srv1335113165=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:52:45,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:52:45,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:52:45,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:52:45,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:52:45,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:52:45,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 20:52:45,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 20:52:45,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 20:52:45,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 20:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 20:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 20:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 20:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 20:52:45,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 20:52:45,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 20:52:45,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 20:52:45,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 20:52:45,770 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:45,770 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 20:52:45,771 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 20:52:46,037 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 439 ms to try 5000 different iterations. Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1788953183648717. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040719614969597, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05811479023446192, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040719614969597, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23916666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:46,079 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1200189448:82 , srv1288883104:82 , srv2130272343:82 , srv2136593238:82 , srv887096966:82 , srv1204379949:83 , srv1481010184:83 , srv1592952573:83 , srv1806400400:83 , srv2039095464:83 , srv2114887160:83 , srv656565098:83 , srv817979368:83 , srv89243407:83 , srv1012263267:84 , srv1110912192:84 , srv1428875004:84 , srv1822091113:84 , srv297464682:84 , srv385475962:84 , srv422277851:84 , srv704063843:84 , srv718126802:84 , srv911678090:84 , srv916237271:84 , srv1147068467:85 , srv1227331566:85 , srv1285576371:85 , srv1335113165:85 , srv1370846652:85 , srv145698554:85 , srv1489560683:85 , srv1504108283:85 , srv2025469200:85 , srv22448346:85 , srv343866517:85 , srv380679175:85 , srv45293674:85 , srv956417765:85 , srv969278174:85 , srv1079923235:86 , srv1204594708:86 , srv121851787:86 , srv1378064232:86 , srv1826127928:86 , srv201962836:86 , srv2077968598:86 , srv316259073:86 , srv33329397:86 , srv373435707:86 , srv645134418:86 , srv664085787:86 , srv891151537:86 , srv1093883575:87 , srv1360142925:87 , srv1624814483:87 , srv1680782224:87 , srv1733294273:87 , srv1760445923:87 , srv1833041721:87 , srv2002048809:87 , srv2032324913:87 , srv268709865:87 , srv323102590:87 , srv813950210:87 , srv833011667:87 , srv846382561:87 , srv1002937901:88 , srv1162560508:88 , srv1569104242:88 , srv1683124142:88 , srv1718143094:88 , srv1936499168:88 , srv316643571:88 , srv416582570:88 , srv672531609:88 , srv1318976506:89 , srv1323186939:89 , srv1436768975:89 , srv1625352951:89 , srv1907616528:89 , srv1989121199:89 , srv1997687412:89 , srv2007167689:89 , srv243499026:89 , srv446364696:89 , srv665925944:89 , srv100676573:90 , srv1040909541:90 , srv1188436843:90 , srv1341430296:90 , srv1344233399:90 , srv1427749919:90 , srv234544888:90 , srv143142669:91 , srv1448644977:91 , srv1716772794:91 , srv1820827041:91 , srv404881017:91 , srv528272546:452 } 2023-09-23 20:52:46,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:52:46,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2114887160=60, srv1683124142=39, srv1716772794=40, srv1227331566=14, srv121851787=13, srv45293674=80, srv1288883104=16, srv2130272343=61, srv1378064232=24, srv1822091113=46, srv145698554=30, srv1427749919=25, srv316643571=69, srv1428875004=26, srv1200189448=10, srv343866517=72, srv373435707=73, srv1989121199=51, srv817979368=90, srv1718143094=41, srv1733294273=42, srv1040909541=3, srv813950210=89, srv1341430296=20, srv1110912192=6, srv1569104242=34, srv833011667=91, srv1448644977=29, srv1360142925=22, srv2039095464=58, srv1079923235=4, srv316259073=68, srv385475962=75, srv1833041721=48, srv201962836=55, srv1323186939=18, srv1489560683=32, srv1680782224=38, srv89243407=95, srv1625352951=37, srv665925944=85, srv1936499168=50, srv2077968598=59, srv645134418=82, srv297464682=67, srv33329397=71, srv100676573=1, srv887096966=93, srv1760445923=43, srv1162560508=8, srv656565098=83, srv323102590=70, srv268709865=66, srv1504108283=33, srv1826127928=47, srv1481010184=31, srv243499026=65, srv1318976506=17, srv422277851=78, srv1204379949=11, srv2032324913=57, srv2002048809=53, srv916237271=97, srv380679175=74, srv2136593238=62, srv416582570=77, srv891151537=94, srv1093883575=5, srv404881017=76, srv1204594708=12, srv234544888=64, srv846382561=92, srv911678090=96, srv1285576371=15, srv143142669=27, srv1436768975=28, srv1806400400=44, srv718126802=88, srv664085787=84, srv2025469200=56, srv1592952573=35, srv1820827041=45, srv22448346=63, srv1344233399=21, srv1188436843=9, srv672531609=86, srv528272546=81, srv1012263267=2, srv1147068467=7, srv1997687412=52, srv704063843=87, srv1370846652=23, srv1907616528=49, srv446364696=79, srv1624814483=36, srv956417765=98, srv2007167689=54, srv969278174=99, srv1002937901=0, srv1335113165=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:52:46,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:52:46,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:52:46,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:52:46,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:52:46,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:52:46,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:52:46,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:52:46,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 20:52:46,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 20:52:46,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 20:52:46,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 20:52:46,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 20:52:46,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 20:52:46,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 20:52:46,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 20:52:46,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 20:52:46,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 20:52:46,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 20:52:46,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 20:52:46,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 20:52:46,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 20:52:46,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 20:52:46,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 20:52:46,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 20:52:46,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 20:52:46,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040719614969597, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05811479023446192, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040719614969597, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23916666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:46,261 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 20:52:46,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1788624922872659, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040719614969597, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05811479023446192, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040719614969597, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23916666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 20:52:46,465 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 297 ms to try 5000 different iterations. Found a solution that moves 2631 regions; Going from a computed imbalance of 0.1788624922872659 to a new imbalance of 0.014858301715127233. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019169045380133934); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029560947332298745, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019169045380133934); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:46,477 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1820827041:80 , srv404881017:80 , srv1040909541:81 , srv1188436843:81 , srv143142669:81 , srv1448644977:81 , srv1989121199:81 , srv1323186939:82 , srv1344233399:82 , srv1427749919:82 , srv1436768975:82 , srv1569104242:82 , srv1625352951:82 , srv1683124142:82 , srv1936499168:82 , srv2007167689:82 , srv416582570:82 , srv672531609:82 , srv1360142925:83 , srv234544888:83 , srv243499026:83 , srv665925944:83 , srv100676573:84 , srv1079923235:84 , srv1093883575:84 , srv1162560508:84 , srv1318976506:84 , srv1833041721:84 , srv1907616528:84 , srv1997687412:84 , srv446364696:84 , srv891151537:84 , srv1002937901:85 , srv1335113165:85 , srv1341430296:85 , srv145698554:85 , srv1718143094:85 , srv1733294273:85 , srv201962836:85 , srv316643571:85 , srv323102590:85 , srv373435707:85 , srv813950210:85 , srv1489560683:86 , srv1624814483:86 , srv1680782224:86 , srv1716772794:86 , srv2025469200:86 , srv22448346:86 , srv343866517:86 , srv833011667:86 , srv1204594708:87 , srv1285576371:87 , srv1370846652:87 , srv1428875004:87 , srv2002048809:87 , srv316259073:87 , srv33329397:87 , srv380679175:87 , srv45293674:87 , srv664085787:87 , srv846382561:87 , srv916237271:87 , srv1110912192:88 , srv1147068467:88 , srv121851787:88 , srv1288883104:88 , srv1378064232:88 , srv1481010184:88 , srv1504108283:88 , srv1760445923:88 , srv1806400400:88 , srv1822091113:88 , srv2032324913:88 , srv2039095464:88 , srv2077968598:88 , srv2136593238:88 , srv268709865:88 , srv645134418:88 , srv704063843:88 , srv718126802:88 , srv887096966:88 , srv911678090:88 , srv956417765:88 , srv1012263267:89 , srv1204379949:89 , srv1826127928:89 , srv297464682:89 , srv385475962:89 , srv89243407:89 , srv1200189448:90 , srv1227331566:90 , srv1592952573:90 , srv2114887160:90 , srv656565098:90 , srv969278174:90 , srv817979368:91 , srv422277851:94 , srv2130272343:95 , srv528272546:492 } 2023-09-23 20:52:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:52:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2114887160=60, srv1683124142=39, srv1716772794=40, srv1227331566=14, srv121851787=13, srv45293674=80, srv1288883104=16, srv2130272343=61, srv1378064232=24, srv1822091113=46, srv145698554=30, srv1427749919=25, srv316643571=69, srv1428875004=26, srv1200189448=10, srv343866517=72, srv373435707=73, srv1989121199=51, srv817979368=90, srv1718143094=41, srv1733294273=42, srv1040909541=3, srv813950210=89, srv1341430296=20, srv1110912192=6, srv1569104242=34, srv833011667=91, srv1448644977=29, srv1360142925=22, srv2039095464=58, srv1079923235=4, srv316259073=68, srv385475962=75, srv1833041721=48, srv201962836=55, srv1323186939=18, srv1489560683=32, srv1680782224=38, srv89243407=95, srv1625352951=37, srv665925944=85, srv1936499168=50, srv2077968598=59, srv645134418=82, srv297464682=67, srv33329397=71, srv100676573=1, srv887096966=93, srv1760445923=43, srv1162560508=8, srv656565098=83, srv323102590=70, srv268709865=66, srv1504108283=33, srv1826127928=47, srv1481010184=31, srv243499026=65, srv1318976506=17, srv422277851=78, srv1204379949=11, srv2032324913=57, srv2002048809=53, srv916237271=97, srv380679175=74, srv2136593238=62, srv416582570=77, srv891151537=94, srv1093883575=5, srv404881017=76, srv1204594708=12, srv234544888=64, srv846382561=92, srv911678090=96, srv1285576371=15, srv143142669=27, srv1436768975=28, srv1806400400=44, srv718126802=88, srv664085787=84, srv2025469200=56, srv1592952573=35, srv1820827041=45, srv22448346=63, srv1344233399=21, srv1188436843=9, srv672531609=86, srv528272546=81, srv1012263267=2, srv1147068467=7, srv1997687412=52, srv704063843=87, srv1370846652=23, srv1907616528=49, srv446364696=79, srv1624814483=36, srv956417765=98, srv2007167689=54, srv969278174=99, srv1002937901=0, srv1335113165=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:52:46,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:52:46,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:52:46,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:52:46,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:52:46,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:52:46,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 20:52:46,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 20:52:46,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 20:52:46,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 20:52:46,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 20:52:46,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 20:52:46,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 20:52:46,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 20:52:46,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 20:52:46,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 20:52:46,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 20:52:46,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 20:52:46,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019169045380133934); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029560947332298745, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019169045380133934); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:46,618 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 20:52:46,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014839879059371916, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019169045380133934); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029560947332298745, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019169045380133934); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 20:52:46,785 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 244 ms to try 5000 different iterations. Found a solution that moves 981 regions; Going from a computed imbalance of 0.014839879059371916 to a new imbalance of 7.293688436369756E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007550800642840709); MoveCostFunction : (multiplier=7.0, imbalance=0.109, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:46,787 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv422277851:79 , srv1716772794:81 , srv2130272343:81 , srv1227331566:82 , srv1341430296:82 , srv1760445923:82 , srv1826127928:82 , srv969278174:82 , srv1624814483:83 , srv2032324913:83 , srv268709865:83 , srv817979368:83 , srv100676573:84 , srv1012263267:84 , srv1147068467:84 , srv1162560508:84 , srv121851787:84 , srv1378064232:84 , srv2002048809:84 , srv2077968598:84 , srv2114887160:84 , srv234544888:84 , srv316259073:84 , srv316643571:84 , srv645134418:84 , srv656565098:84 , srv664085787:84 , srv846382561:84 , srv1002937901:85 , srv1200189448:85 , srv1204379949:85 , srv1204594708:85 , srv1285576371:85 , srv1318976506:85 , srv1344233399:85 , srv1370846652:85 , srv1427749919:85 , srv1448644977:85 , srv1504108283:85 , srv1592952573:85 , srv1680782224:85 , srv1718143094:85 , srv22448346:85 , srv297464682:85 , srv323102590:85 , srv33329397:85 , srv385475962:85 , srv446364696:85 , srv45293674:85 , srv718126802:85 , srv813950210:85 , srv833011667:85 , srv89243407:85 , srv911678090:85 , srv956417765:85 , srv1040909541:86 , srv1093883575:86 , srv1110912192:86 , srv1323186939:86 , srv1428875004:86 , srv143142669:86 , srv1436768975:86 , srv1481010184:86 , srv1625352951:86 , srv1733294273:86 , srv1806400400:86 , srv1822091113:86 , srv1833041721:86 , srv1907616528:86 , srv1997687412:86 , srv2007167689:86 , srv201962836:86 , srv2039095464:86 , srv243499026:86 , srv380679175:86 , srv404881017:86 , srv665925944:86 , srv704063843:86 , srv916237271:86 , srv1079923235:87 , srv1188436843:87 , srv145698554:87 , srv1489560683:87 , srv1569104242:87 , srv1683124142:87 , srv1820827041:87 , srv1936499168:87 , srv2025469200:87 , srv2136593238:87 , srv343866517:87 , srv416582570:87 , srv672531609:87 , srv1288883104:88 , srv1335113165:88 , srv1360142925:88 , srv1989121199:88 , srv373435707:88 , srv887096966:88 , srv891151537:89 , srv528272546:566 } 2023-09-23 20:52:46,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:52:46,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2114887160=60, srv1683124142=39, srv1716772794=40, srv1227331566=14, srv121851787=13, srv45293674=80, srv1288883104=16, srv2130272343=61, srv1378064232=24, srv1822091113=46, srv145698554=30, srv1427749919=25, srv316643571=69, srv1428875004=26, srv1200189448=10, srv343866517=72, srv373435707=73, srv1989121199=51, srv817979368=90, srv1718143094=41, srv1733294273=42, srv1040909541=3, srv813950210=89, srv1341430296=20, srv1110912192=6, srv1569104242=34, srv833011667=91, srv1448644977=29, srv1360142925=22, srv2039095464=58, srv1079923235=4, srv316259073=68, srv385475962=75, srv1833041721=48, srv201962836=55, srv1323186939=18, srv1489560683=32, srv1680782224=38, srv89243407=95, srv1625352951=37, srv665925944=85, srv1936499168=50, srv2077968598=59, srv645134418=82, srv297464682=67, srv33329397=71, srv100676573=1, srv887096966=93, srv1760445923=43, srv1162560508=8, srv656565098=83, srv323102590=70, srv268709865=66, srv1504108283=33, srv1826127928=47, srv1481010184=31, srv243499026=65, srv1318976506=17, srv422277851=78, srv1204379949=11, srv2032324913=57, srv2002048809=53, srv916237271=97, srv380679175=74, srv2136593238=62, srv416582570=77, srv891151537=94, srv1093883575=5, srv404881017=76, srv1204594708=12, srv234544888=64, srv846382561=92, srv911678090=96, srv1285576371=15, srv143142669=27, srv1436768975=28, srv1806400400=44, srv718126802=88, srv664085787=84, srv2025469200=56, srv1592952573=35, srv1820827041=45, srv22448346=63, srv1344233399=21, srv1188436843=9, srv672531609=86, srv528272546=81, srv1012263267=2, srv1147068467=7, srv1997687412=52, srv704063843=87, srv1370846652=23, srv1907616528=49, srv446364696=79, srv1624814483=36, srv956417765=98, srv2007167689=54, srv969278174=99, srv1002937901=0, srv1335113165=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:52:46,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:52:46,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:52:46,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:52:46,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:52:46,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:52:46,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:52:46,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:52:46,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 20:52:46,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 20:52:46,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 20:52:46,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 20:52:46,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 20:52:46,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 20:52:46,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 20:52:46,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 20:52:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 20:52:46,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 20:52:46,905 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007550800642840709); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:46,905 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 20:52:46,905 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.224997348205689E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007550800642840709); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 20:52:47,137 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 311 ms to try 5000 different iterations. Found a solution that moves 254 regions; Going from a computed imbalance of 7.224997348205689E-4 to a new imbalance of 1.994599783590222E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:47,138 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1360142925:83 , srv1907616528:83 , srv1997687412:83 , srv373435707:83 , srv833011667:83 , srv891151537:83 , srv1002937901:84 , srv100676573:84 , srv1012263267:84 , srv1093883575:84 , srv1110912192:84 , srv1188436843:84 , srv121851787:84 , srv1318976506:84 , srv1335113165:84 , srv1370846652:84 , srv1378064232:84 , srv1427749919:84 , srv143142669:84 , srv1489560683:84 , srv1504108283:84 , srv1592952573:84 , srv1625352951:84 , srv1680782224:84 , srv1716772794:84 , srv1718143094:84 , srv1733294273:84 , srv1820827041:84 , srv1822091113:84 , srv1989121199:84 , srv2002048809:84 , srv2032324913:84 , srv2077968598:84 , srv2130272343:84 , srv234544888:84 , srv243499026:84 , srv297464682:84 , srv316643571:84 , srv323102590:84 , srv33329397:84 , srv343866517:84 , srv385475962:84 , srv446364696:84 , srv645134418:84 , srv665925944:84 , srv704063843:84 , srv846382561:84 , srv887096966:84 , srv956417765:84 , srv1079923235:85 , srv1147068467:85 , srv1200189448:85 , srv1204379949:85 , srv1204594708:85 , srv1288883104:85 , srv1323186939:85 , srv1341430296:85 , srv1344233399:85 , srv1428875004:85 , srv1436768975:85 , srv1448644977:85 , srv145698554:85 , srv1481010184:85 , srv1569104242:85 , srv1683124142:85 , srv1760445923:85 , srv1806400400:85 , srv1826127928:85 , srv1833041721:85 , srv1936499168:85 , srv2007167689:85 , srv201962836:85 , srv2025469200:85 , srv2039095464:85 , srv2114887160:85 , srv2136593238:85 , srv268709865:85 , srv316259073:85 , srv380679175:85 , srv404881017:85 , srv416582570:85 , srv422277851:85 , srv45293674:85 , srv656565098:85 , srv664085787:85 , srv672531609:85 , srv718126802:85 , srv813950210:85 , srv817979368:85 , srv89243407:85 , srv911678090:85 , srv916237271:85 , srv1040909541:86 , srv1162560508:86 , srv1227331566:86 , srv1285576371:86 , srv22448346:86 , srv969278174:86 , srv1624814483:87 , srv528272546:632 } 2023-09-23 20:52:47,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:52:47,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2114887160=60, srv1683124142=39, srv1716772794=40, srv1227331566=14, srv121851787=13, srv45293674=80, srv1288883104=16, srv2130272343=61, srv1378064232=24, srv1822091113=46, srv145698554=30, srv1427749919=25, srv316643571=69, srv1428875004=26, srv1200189448=10, srv343866517=72, srv373435707=73, srv1989121199=51, srv817979368=90, srv1718143094=41, srv1733294273=42, srv1040909541=3, srv813950210=89, srv1341430296=20, srv1110912192=6, srv1569104242=34, srv833011667=91, srv1448644977=29, srv1360142925=22, srv2039095464=58, srv1079923235=4, srv316259073=68, srv385475962=75, srv1833041721=48, srv201962836=55, srv1323186939=18, srv1489560683=32, srv1680782224=38, srv89243407=95, srv1625352951=37, srv665925944=85, srv1936499168=50, srv2077968598=59, srv645134418=82, srv297464682=67, srv33329397=71, srv100676573=1, srv887096966=93, srv1760445923=43, srv1162560508=8, srv656565098=83, srv323102590=70, srv268709865=66, srv1504108283=33, srv1826127928=47, srv1481010184=31, srv243499026=65, srv1318976506=17, srv422277851=78, srv1204379949=11, srv2032324913=57, srv2002048809=53, srv916237271=97, srv380679175=74, srv2136593238=62, srv416582570=77, srv891151537=94, srv1093883575=5, srv404881017=76, srv1204594708=12, srv234544888=64, srv846382561=92, srv911678090=96, srv1285576371=15, srv143142669=27, srv1436768975=28, srv1806400400=44, srv718126802=88, srv664085787=84, srv2025469200=56, srv1592952573=35, srv1820827041=45, srv22448346=63, srv1344233399=21, srv1188436843=9, srv672531609=86, srv528272546=81, srv1012263267=2, srv1147068467=7, srv1997687412=52, srv704063843=87, srv1370846652=23, srv1907616528=49, srv446364696=79, srv1624814483=36, srv956417765=98, srv2007167689=54, srv969278174=99, srv1002937901=0, srv1335113165=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:52:47,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:52:47,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:52:47,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 20:52:47,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 20:52:47,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 20:52:47,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 20:52:47,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 20:52:47,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 20:52:47,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 20:52:47,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 20:52:47,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 20:52:47,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 20:52:47,251 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.816745182227604E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:47,251 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1360142925:83 , srv1907616528:83 , srv1997687412:83 , srv373435707:83 , srv833011667:83 , srv891151537:83 , srv1002937901:84 , srv100676573:84 , srv1012263267:84 , srv1093883575:84 , srv1110912192:84 , srv1188436843:84 , srv121851787:84 , srv1318976506:84 , srv1335113165:84 , srv1370846652:84 , srv1378064232:84 , srv1427749919:84 , srv143142669:84 , srv1489560683:84 , srv1504108283:84 , srv1592952573:84 , srv1625352951:84 , srv1680782224:84 , srv1716772794:84 , srv1718143094:84 , srv1733294273:84 , srv1820827041:84 , srv1822091113:84 , srv1989121199:84 , srv2002048809:84 , srv2032324913:84 , srv2077968598:84 , srv2130272343:84 , srv234544888:84 , srv243499026:84 , srv297464682:84 , srv316643571:84 , srv323102590:84 , srv33329397:84 , srv343866517:84 , srv385475962:84 , srv446364696:84 , srv645134418:84 , srv665925944:84 , srv704063843:84 , srv846382561:84 , srv887096966:84 , srv956417765:84 , srv1079923235:85 , srv1147068467:85 , srv1200189448:85 , srv1204379949:85 , srv1204594708:85 , srv1288883104:85 , srv1323186939:85 , srv1341430296:85 , srv1344233399:85 , srv1428875004:85 , srv1436768975:85 , srv1448644977:85 , srv145698554:85 , srv1481010184:85 , srv1569104242:85 , srv1683124142:85 , srv1760445923:85 , srv1806400400:85 , srv1826127928:85 , srv1833041721:85 , srv1936499168:85 , srv2007167689:85 , srv201962836:85 , srv2025469200:85 , srv2039095464:85 , srv2114887160:85 , srv2136593238:85 , srv268709865:85 , srv316259073:85 , srv380679175:85 , srv404881017:85 , srv416582570:85 , srv422277851:85 , srv45293674:85 , srv656565098:85 , srv664085787:85 , srv672531609:85 , srv718126802:85 , srv813950210:85 , srv817979368:85 , srv89243407:85 , srv911678090:85 , srv916237271:85 , srv1040909541:86 , srv1162560508:86 , srv1227331566:86 , srv1285576371:86 , srv22448346:86 , srv969278174:86 , srv1624814483:87 , srv528272546:632 } 2023-09-23 20:52:47,306 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 321) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 175), AvailableMemoryMB=21961 (was 22595) 2023-09-23 20:52:47,316 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=173, AvailableMemoryMB=21961 2023-09-23 20:52:47,317 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 20:52:47,317 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-23 20:52:47,328 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1069190974:84 , srv123899832:84 , srv1452955514:84 , srv1514015417:84 , srv1519318258:84 , srv1522407248:84 , srv1551417228:84 , srv1564168463:84 , srv1636318832:84 , srv1649802268:84 , srv1741836146:84 , srv1772834145:84 , srv1852949264:84 , srv1889679710:84 , srv1988887901:84 , srv2048368887:84 , srv2065969670:84 , srv2118641956:84 , srv265563309:84 , srv274479860:84 , srv295217629:84 , srv349379382:84 , srv381895044:84 , srv438342060:84 , srv511978402:84 , srv57171965:84 , srv58881974:84 , srv929149585:84 , srv950376315:84 , srv724200987:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-23 20:52:47,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:52:47,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1772834145=11, srv438342060=23, srv1551417228=6, srv2118641956=17, srv1988887901=14, srv274479860=19, srv381895044=22, srv349379382=21, srv295217629=20, srv1852949264=12, srv2065969670=16, srv724200987=27, srv2048368887=15, srv1636318832=8, srv511978402=24, srv1452955514=2, srv123899832=1, srv1889679710=13, srv1514015417=3, srv57171965=25, srv929149585=28, srv265563309=18, srv58881974=26, srv950376315=29, srv1649802268=9, srv1519318258=4, srv1069190974=0, srv1564168463=7, srv1741836146=10, srv1522407248=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:52:47,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:52:47,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:52:47,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:52:47,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:52:47,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:52:47,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:52:47,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:52:47,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:52:47,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:52:47,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:52:47,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-23 20:52:47,371 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.11888720149211, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:52:47,371 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912567652954157, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.11888720149211, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-23 20:54:47,349 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9748863 different iterations. Found a solution that moves 1836 regions; Going from a computed imbalance of 0.9912567652954157 to a new imbalance of 4.2853155918867096E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.68, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:54:47,354 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1069190974:90 , srv123899832:90 , srv1452955514:90 , srv1514015417:90 , srv1519318258:90 , srv1522407248:90 , srv1551417228:90 , srv1564168463:90 , srv1636318832:90 , srv1649802268:90 , srv1741836146:90 , srv1772834145:90 , srv1852949264:90 , srv1889679710:90 , srv1988887901:90 , srv2048368887:90 , srv2065969670:90 , srv2118641956:90 , srv265563309:90 , srv274479860:90 , srv295217629:90 , srv349379382:90 , srv381895044:90 , srv438342060:90 , srv511978402:90 , srv57171965:90 , srv58881974:90 , srv724200987:90 , srv929149585:90 , srv950376315:90 } 2023-09-23 20:54:47,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 20:54:47,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1772834145=11, srv438342060=23, srv1551417228=6, srv2118641956=17, srv1988887901=14, srv274479860=19, srv381895044=22, srv349379382=21, srv295217629=20, srv1852949264=12, srv2065969670=16, srv724200987=27, srv2048368887=15, srv1636318832=8, srv511978402=24, srv1452955514=2, srv123899832=1, srv1889679710=13, srv1514015417=3, srv57171965=25, srv929149585=28, srv265563309=18, srv58881974=26, srv950376315=29, srv1649802268=9, srv1519318258=4, srv1069190974=0, srv1564168463=7, srv1741836146=10, srv1522407248=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 20:54:47,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 20:54:47,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 20:54:47,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 20:54:47,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 20:54:47,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 20:54:47,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 20:54:47,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 20:54:47,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 20:54:47,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 20:54:47,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 20:54:47,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-23 20:54:47,369 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 20:54:47,369 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1069190974:90 , srv123899832:90 , srv1452955514:90 , srv1514015417:90 , srv1519318258:90 , srv1522407248:90 , srv1551417228:90 , srv1564168463:90 , srv1636318832:90 , srv1649802268:90 , srv1741836146:90 , srv1772834145:90 , srv1852949264:90 , srv1889679710:90 , srv1988887901:90 , srv2048368887:90 , srv2065969670:90 , srv2118641956:90 , srv265563309:90 , srv274479860:90 , srv295217629:90 , srv349379382:90 , srv381895044:90 , srv438342060:90 , srv511978402:90 , srv57171965:90 , srv58881974:90 , srv724200987:90 , srv929149585:90 , srv950376315:90 } 2023-09-23 20:54:47,380 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=154 (was 344), ProcessCount=174 (was 173) - ProcessCount LEAK? -, AvailableMemoryMB=21113 (was 21961)