2023-02-11 19:55:58,326 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-11 19:55:58,651 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 19:55:58,761 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 19:55:58,824 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-11 19:55:58,830 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 19:55:58,843 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=209, ProcessCount=168, AvailableMemoryMB=4213 2023-02-11 19:55:58,845 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-11 19:55:58,845 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 19:55:58,972 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009593933:84 , srv1033033887:84 , srv1045669613:84 , srv1050849155:84 , srv1075125727:84 , srv1145582048:84 , srv1146915614:84 , srv1176457946:84 , srv1185871978:84 , srv1189823533:84 , srv1214233338:84 , srv1216745527:84 , srv1266784275:84 , srv1269832296:84 , srv1283958938:84 , srv1290155068:84 , srv130927963:84 , srv1317837699:84 , srv1358370848:84 , srv1364533382:84 , srv1367807547:84 , srv1380014758:84 , srv1401498991:84 , srv1403871113:84 , srv1444684264:84 , srv1457854849:84 , srv1476954640:84 , srv1483193548:84 , srv1485681200:84 , srv1502761281:84 , srv150730910:84 , srv1538548986:84 , srv1539323434:84 , srv1544494490:84 , srv1545256365:84 , srv1588577461:84 , srv1643098520:84 , srv1646042753:84 , srv1653092175:84 , srv1675609273:84 , srv1734446232:84 , srv1748352628:84 , srv1754152942:84 , srv1761415484:84 , srv1795548121:84 , srv1822949218:84 , srv1837205862:84 , srv1840561370:84 , srv1854114441:84 , srv190746759:84 , srv1934616135:84 , srv1958700076:84 , srv2016925776:84 , srv2060890343:84 , srv2074961396:84 , srv2121911234:84 , srv2146562111:84 , srv214681506:84 , srv219763387:84 , srv284501302:84 , srv298191604:84 , srv309244707:84 , srv319145232:84 , srv354130561:84 , srv373833666:84 , srv399584438:84 , srv449696393:84 , srv522925931:84 , srv534409875:84 , srv548066447:84 , srv583518834:84 , srv599894188:84 , srv623261566:84 , srv624262984:84 , srv634663475:84 , srv654533434:84 , srv680805238:84 , srv694491579:84 , srv710404478:84 , srv720449578:84 , srv722937079:84 , srv725006819:84 , srv728694941:84 , srv730964212:84 , srv757296558:84 , srv807311457:84 , srv831426590:84 , srv841038898:84 , srv84908664:84 , srv864099257:84 , srv865406278:84 , srv873534218:84 , srv875306744:84 , srv876929712:84 , srv880130260:84 , srv928727869:84 , srv930238814:84 , srv941293933:84 , srv998760019:84 , srv1061397530:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-11 19:55:58,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:55:58,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373833666=65, srv728694941=83, srv1380014758=22, srv1646042753=38, srv880130260=95, srv1146915614=7, srv150730910=31, srv875306744=93, srv354130561=64, srv1476954640=27, srv1653092175=39, srv1364533382=20, srv2016925776=53, srv2121911234=56, srv1214233338=11, srv219763387=59, srv599894188=72, srv710404478=79, srv1045669613=2, srv654533434=76, srv1539323434=33, srv1185871978=9, srv298191604=61, srv1545256365=35, srv522925931=68, srv1290155068=16, srv1837205862=47, srv720449578=80, srv930238814=97, srv399584438=66, srv1283958938=15, srv1544494490=34, srv1367807547=21, srv831426590=87, srv928727869=96, srv1444684264=25, srv449696393=67, srv680805238=77, srv1075125727=5, srv757296558=85, srv864099257=90, srv1748352628=42, srv1761415484=44, srv876929712=94, srv1033033887=1, srv309244707=62, srv841038898=88, srv1588577461=36, srv2146562111=57, srv1358370848=19, srv1502761281=30, srv1189823533=10, srv1317837699=18, srv865406278=91, srv722937079=81, srv1934616135=51, srv1840561370=48, srv998760019=99, srv1485681200=29, srv1822949218=46, srv319145232=63, srv1266784275=13, srv1269832296=14, srv1216745527=12, srv624262984=74, srv634663475=75, srv807311457=86, srv534409875=69, srv1643098520=37, srv84908664=89, srv1754152942=43, srv1403871113=24, srv190746759=50, srv1958700076=52, srv1795548121=45, srv873534218=92, srv694491579=78, srv130927963=17, srv583518834=71, srv548066447=70, srv284501302=60, srv1176457946=8, srv623261566=73, srv1538548986=32, srv1401498991=23, srv1009593933=0, srv1675609273=40, srv214681506=58, srv1061397530=4, srv1854114441=49, srv2074961396=55, srv1483193548=28, srv1734446232=41, srv1050849155=3, srv2060890343=54, srv1145582048=6, srv725006819=82, srv730964212=84, srv941293933=98, srv1457854849=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-11 19:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-11 19:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-11 19:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-11 19:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-11 19:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-11 19:55:59,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-11 19:55:59,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-11 19:55:59,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-11 19:55:59,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-11 19:55:59,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-11 19:55:59,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-11 19:55:59,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-11 19:55:59,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:55:59,259 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-11 19:55:59,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-11 19:55:59,560 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 459 ms to try 5000 different iterations. Found a solution that moves 4692 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17850305615499898. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04142191490375056, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0568811547772995, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5213333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04142191490375056, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23733333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:55:59,582 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv298191604:80 , srv880130260:80 , srv522925931:81 , srv1761415484:82 , srv354130561:82 , srv694491579:82 , srv1269832296:83 , srv1485681200:83 , srv1958700076:83 , srv449696393:83 , srv634663475:83 , srv722937079:83 , srv807311457:83 , srv84908664:83 , srv875306744:83 , srv1216745527:84 , srv130927963:84 , srv1457854849:84 , srv1502761281:84 , srv1643098520:84 , srv1653092175:84 , srv1837205862:84 , srv319145232:84 , srv583518834:84 , srv654533434:84 , srv728694941:84 , srv730964212:84 , srv1075125727:85 , srv1266784275:85 , srv1290155068:85 , srv1317837699:85 , srv1358370848:85 , srv1401498991:85 , srv1646042753:85 , srv1675609273:85 , srv1822949218:85 , srv2060890343:85 , srv2074961396:85 , srv548066447:85 , srv757296558:85 , srv1045669613:86 , srv1367807547:86 , srv1380014758:86 , srv1476954640:86 , srv150730910:86 , srv1538548986:86 , srv1748352628:86 , srv214681506:86 , srv219763387:86 , srv725006819:86 , srv841038898:86 , srv873534218:86 , srv998760019:86 , srv1145582048:87 , srv1146915614:87 , srv1185871978:87 , srv1189823533:87 , srv1283958938:87 , srv1444684264:87 , srv1539323434:87 , srv1754152942:87 , srv190746759:87 , srv1934616135:87 , srv2016925776:87 , srv284501302:87 , srv309244707:87 , srv373833666:87 , srv534409875:87 , srv599894188:87 , srv680805238:87 , srv930238814:87 , srv1009593933:88 , srv1033033887:88 , srv1050849155:88 , srv1364533382:88 , srv1544494490:88 , srv1734446232:88 , srv2146562111:88 , srv623261566:88 , srv624262984:88 , srv831426590:88 , srv941293933:88 , srv1588577461:89 , srv1795548121:89 , srv1840561370:89 , srv399584438:89 , srv864099257:89 , srv865406278:89 , srv876929712:89 , srv928727869:89 , srv1854114441:90 , srv710404478:90 , srv1176457946:92 , srv1214233338:92 , srv2121911234:92 , srv1403871113:93 , srv720449578:93 , srv1545256365:94 , srv1483193548:95 , srv1061397530:458 } 2023-02-11 19:55:59,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:55:59,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373833666=65, srv728694941=83, srv1380014758=22, srv1646042753=38, srv880130260=95, srv1146915614=7, srv150730910=31, srv875306744=93, srv354130561=64, srv1476954640=27, srv1653092175=39, srv1364533382=20, srv2016925776=53, srv2121911234=56, srv1214233338=11, srv219763387=59, srv599894188=72, srv710404478=79, srv1045669613=2, srv654533434=76, srv1539323434=33, srv1185871978=9, srv298191604=61, srv1545256365=35, srv522925931=68, srv1290155068=16, srv1837205862=47, srv720449578=80, srv930238814=97, srv399584438=66, srv1283958938=15, srv1544494490=34, srv1367807547=21, srv831426590=87, srv928727869=96, srv1444684264=25, srv449696393=67, srv680805238=77, srv1075125727=5, srv757296558=85, srv864099257=90, srv1748352628=42, srv1761415484=44, srv876929712=94, srv1033033887=1, srv309244707=62, srv841038898=88, srv1588577461=36, srv2146562111=57, srv1358370848=19, srv1502761281=30, srv1189823533=10, srv1317837699=18, srv865406278=91, srv722937079=81, srv1934616135=51, srv1840561370=48, srv998760019=99, srv1485681200=29, srv1822949218=46, srv319145232=63, srv1266784275=13, srv1269832296=14, srv1216745527=12, srv624262984=74, srv634663475=75, srv807311457=86, srv534409875=69, srv1643098520=37, srv84908664=89, srv1754152942=43, srv1403871113=24, srv190746759=50, srv1958700076=52, srv1795548121=45, srv873534218=92, srv694491579=78, srv130927963=17, srv583518834=71, srv548066447=70, srv284501302=60, srv1176457946=8, srv623261566=73, srv1538548986=32, srv1401498991=23, srv1009593933=0, srv1675609273=40, srv214681506=58, srv1061397530=4, srv1854114441=49, srv2074961396=55, srv1483193548=28, srv1734446232=41, srv1050849155=3, srv2060890343=54, srv1145582048=6, srv725006819=82, srv730964212=84, srv941293933=98, srv1457854849=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:55:59,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:55:59,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:55:59,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:55:59,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-11 19:55:59,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-11 19:55:59,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-11 19:55:59,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-11 19:55:59,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-11 19:55:59,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-11 19:55:59,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-11 19:55:59,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-11 19:55:59,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-11 19:55:59,751 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04142191490375056, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0568811547772995, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04142191490375056, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23733333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:55:59,752 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-11 19:55:59,752 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17847020206879452, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04142191490375056, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0568811547772995, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04142191490375056, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23733333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-11 19:56:00,077 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 424 ms to try 5000 different iterations. Found a solution that moves 2581 regions; Going from a computed imbalance of 0.17847020206879452 to a new imbalance of 0.015687140118217504. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022038522429637972); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03903148460055621, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2867777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022038522429637972); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:00,093 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1483193548:78 , srv1403871113:79 , srv720449578:79 , srv1176457946:80 , srv1214233338:80 , srv1545256365:80 , srv2121911234:81 , srv1009593933:82 , srv1588577461:82 , srv710404478:82 , srv876929712:82 , srv1033033887:83 , srv1734446232:83 , srv2016925776:83 , srv373833666:83 , srv831426590:83 , srv865406278:83 , srv928727869:83 , srv1050849155:84 , srv1189823533:84 , srv1364533382:84 , srv1795548121:84 , srv1840561370:84 , srv190746759:84 , srv1934616135:84 , srv214681506:84 , srv309244707:84 , srv399584438:84 , srv599894188:84 , srv623261566:84 , srv624262984:84 , srv864099257:84 , srv1045669613:85 , srv1266784275:85 , srv1317837699:85 , srv1367807547:85 , srv1538548986:85 , srv1854114441:85 , srv2146562111:85 , srv873534218:85 , srv930238814:85 , srv998760019:85 , srv1145582048:86 , srv1146915614:86 , srv1185871978:86 , srv1290155068:86 , srv1401498991:86 , srv1457854849:86 , srv1539323434:86 , srv1544494490:86 , srv1646042753:86 , srv1748352628:86 , srv1754152942:86 , srv219763387:86 , srv284501302:86 , srv534409875:86 , srv680805238:86 , srv725006819:86 , srv1075125727:87 , srv1283958938:87 , srv1358370848:87 , srv1380014758:87 , srv1444684264:87 , srv1476954640:87 , srv1502761281:87 , srv150730910:87 , srv2060890343:87 , srv548066447:87 , srv654533434:87 , srv722937079:87 , srv841038898:87 , srv84908664:87 , srv941293933:87 , srv130927963:88 , srv1643098520:88 , srv1653092175:88 , srv1675609273:88 , srv1822949218:88 , srv1958700076:88 , srv2074961396:88 , srv449696393:88 , srv634663475:88 , srv728694941:88 , srv730964212:88 , srv757296558:88 , srv1269832296:89 , srv1485681200:89 , srv1761415484:89 , srv319145232:89 , srv522925931:89 , srv807311457:89 , srv1837205862:90 , srv354130561:90 , srv583518834:90 , srv694491579:90 , srv1216745527:91 , srv875306744:91 , srv880130260:91 , srv298191604:92 , srv1061397530:512 } 2023-02-11 19:56:00,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:56:00,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373833666=65, srv728694941=83, srv1380014758=22, srv1646042753=38, srv880130260=95, srv1146915614=7, srv150730910=31, srv875306744=93, srv354130561=64, srv1476954640=27, srv1653092175=39, srv1364533382=20, srv2016925776=53, srv2121911234=56, srv1214233338=11, srv219763387=59, srv599894188=72, srv710404478=79, srv1045669613=2, srv654533434=76, srv1539323434=33, srv1185871978=9, srv298191604=61, srv1545256365=35, srv522925931=68, srv1290155068=16, srv1837205862=47, srv720449578=80, srv930238814=97, srv399584438=66, srv1283958938=15, srv1544494490=34, srv1367807547=21, srv831426590=87, srv928727869=96, srv1444684264=25, srv449696393=67, srv680805238=77, srv1075125727=5, srv757296558=85, srv864099257=90, srv1748352628=42, srv1761415484=44, srv876929712=94, srv1033033887=1, srv309244707=62, srv841038898=88, srv1588577461=36, srv2146562111=57, srv1358370848=19, srv1502761281=30, srv1189823533=10, srv1317837699=18, srv865406278=91, srv722937079=81, srv1934616135=51, srv1840561370=48, srv998760019=99, srv1485681200=29, srv1822949218=46, srv319145232=63, srv1266784275=13, srv1269832296=14, srv1216745527=12, srv624262984=74, srv634663475=75, srv807311457=86, srv534409875=69, srv1643098520=37, srv84908664=89, srv1754152942=43, srv1403871113=24, srv190746759=50, srv1958700076=52, srv1795548121=45, srv873534218=92, srv694491579=78, srv130927963=17, srv583518834=71, srv548066447=70, srv284501302=60, srv1176457946=8, srv623261566=73, srv1538548986=32, srv1401498991=23, srv1009593933=0, srv1675609273=40, srv214681506=58, srv1061397530=4, srv1854114441=49, srv2074961396=55, srv1483193548=28, srv1734446232=41, srv1050849155=3, srv2060890343=54, srv1145582048=6, srv725006819=82, srv730964212=84, srv941293933=98, srv1457854849=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:56:00,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:56:00,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:56:00,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-11 19:56:00,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-11 19:56:00,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-11 19:56:00,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-11 19:56:00,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-11 19:56:00,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-11 19:56:00,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-11 19:56:00,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-11 19:56:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-11 19:56:00,249 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022038522429637972); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03903148460055621, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022038522429637972); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:00,250 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-11 19:56:00,250 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015669067569945183, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022038522429637972); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03903148460055621, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022038522429637972); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-11 19:56:00,525 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 376 ms to try 5000 different iterations. Found a solution that moves 1000 regions; Going from a computed imbalance of 0.015669067569945183 to a new imbalance of 8.164874065628408E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007901059201196417); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009709587750058922); MoveCostFunction : (multiplier=7.0, imbalance=0.1111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007901059201196417); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:00,526 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1216745527:82 , srv1854114441:82 , srv1145582048:83 , srv1283958938:83 , srv1444684264:83 , srv1483193548:83 , srv1545256365:83 , srv2074961396:83 , srv583518834:83 , srv875306744:83 , srv941293933:83 , srv1146915614:84 , srv1380014758:84 , srv1476954640:84 , srv150730910:84 , srv1544494490:84 , srv1754152942:84 , srv1822949218:84 , srv1837205862:84 , srv1840561370:84 , srv2121911234:84 , srv2146562111:84 , srv219763387:84 , srv319145232:84 , srv680805238:84 , srv841038898:84 , srv1050849155:85 , srv1185871978:85 , srv1189823533:85 , srv1269832296:85 , srv130927963:85 , srv1364533382:85 , srv1485681200:85 , srv1643098520:85 , srv1675609273:85 , srv1748352628:85 , srv1795548121:85 , srv284501302:85 , srv298191604:85 , srv534409875:85 , srv548066447:85 , srv623261566:85 , srv624262984:85 , srv694491579:85 , srv710404478:85 , srv720449578:85 , srv730964212:85 , srv807311457:85 , srv864099257:85 , srv865406278:85 , srv876929712:85 , srv880130260:85 , srv1009593933:86 , srv1033033887:86 , srv1045669613:86 , srv1075125727:86 , srv1176457946:86 , srv1290155068:86 , srv1317837699:86 , srv1358370848:86 , srv1367807547:86 , srv1401498991:86 , srv1403871113:86 , srv1457854849:86 , srv1539323434:86 , srv1588577461:86 , srv1653092175:86 , srv1761415484:86 , srv1934616135:86 , srv2060890343:86 , srv354130561:86 , srv399584438:86 , srv449696393:86 , srv522925931:86 , srv599894188:86 , srv634663475:86 , srv654533434:86 , srv725006819:86 , srv728694941:86 , srv757296558:86 , srv831426590:86 , srv873534218:86 , srv930238814:86 , srv998760019:86 , srv1214233338:87 , srv1502761281:87 , srv1646042753:87 , srv1734446232:87 , srv190746759:87 , srv1958700076:87 , srv2016925776:87 , srv309244707:87 , srv373833666:87 , srv722937079:87 , srv84908664:87 , srv928727869:87 , srv1266784275:88 , srv1538548986:88 , srv214681506:89 , srv1061397530:558 } 2023-02-11 19:56:00,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:56:00,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373833666=65, srv728694941=83, srv1380014758=22, srv1646042753=38, srv880130260=95, srv1146915614=7, srv150730910=31, srv875306744=93, srv354130561=64, srv1476954640=27, srv1653092175=39, srv1364533382=20, srv2016925776=53, srv2121911234=56, srv1214233338=11, srv219763387=59, srv599894188=72, srv710404478=79, srv1045669613=2, srv654533434=76, srv1539323434=33, srv1185871978=9, srv298191604=61, srv1545256365=35, srv522925931=68, srv1290155068=16, srv1837205862=47, srv720449578=80, srv930238814=97, srv399584438=66, srv1283958938=15, srv1544494490=34, srv1367807547=21, srv831426590=87, srv928727869=96, srv1444684264=25, srv449696393=67, srv680805238=77, srv1075125727=5, srv757296558=85, srv864099257=90, srv1748352628=42, srv1761415484=44, srv876929712=94, srv1033033887=1, srv309244707=62, srv841038898=88, srv1588577461=36, srv2146562111=57, srv1358370848=19, srv1502761281=30, srv1189823533=10, srv1317837699=18, srv865406278=91, srv722937079=81, srv1934616135=51, srv1840561370=48, srv998760019=99, srv1485681200=29, srv1822949218=46, srv319145232=63, srv1266784275=13, srv1269832296=14, srv1216745527=12, srv624262984=74, srv634663475=75, srv807311457=86, srv534409875=69, srv1643098520=37, srv84908664=89, srv1754152942=43, srv1403871113=24, srv190746759=50, srv1958700076=52, srv1795548121=45, srv873534218=92, srv694491579=78, srv130927963=17, srv583518834=71, srv548066447=70, srv284501302=60, srv1176457946=8, srv623261566=73, srv1538548986=32, srv1401498991=23, srv1009593933=0, srv1675609273=40, srv214681506=58, srv1061397530=4, srv1854114441=49, srv2074961396=55, srv1483193548=28, srv1734446232=41, srv1050849155=3, srv2060890343=54, srv1145582048=6, srv725006819=82, srv730964212=84, srv941293933=98, srv1457854849=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:56:00,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:56:00,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:56:00,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:56:00,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:56:00,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:56:00,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:56:00,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-11 19:56:00,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-11 19:56:00,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-11 19:56:00,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-11 19:56:00,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-11 19:56:00,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-11 19:56:00,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-11 19:56:00,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-11 19:56:00,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-11 19:56:00,677 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007901059201196417); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009709587750058922); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007901059201196417); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:00,678 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-11 19:56:00,678 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.094852569028951E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007901059201196417); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009709587750058922); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007901059201196417); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-11 19:56:00,943 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 368 ms to try 5000 different iterations. Found a solution that moves 315 regions; Going from a computed imbalance of 8.094852569028951E-4 to a new imbalance of 3.891644569748399E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.035, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:00,944 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1538548986:82 , srv1214233338:83 , srv1539323434:83 , srv1675609273:83 , srv190746759:83 , srv214681506:83 , srv284501302:83 , srv399584438:83 , srv757296558:83 , srv928727869:83 , srv1050849155:84 , srv1176457946:84 , srv1185871978:84 , srv1266784275:84 , srv1290155068:84 , srv1358370848:84 , srv1364533382:84 , srv1403871113:84 , srv1502761281:84 , srv150730910:84 , srv1646042753:84 , srv1653092175:84 , srv1795548121:84 , srv1840561370:84 , srv1958700076:84 , srv2060890343:84 , srv309244707:84 , srv354130561:84 , srv534409875:84 , srv599894188:84 , srv728694941:84 , srv941293933:84 , srv1009593933:85 , srv1033033887:85 , srv1045669613:85 , srv1075125727:85 , srv1146915614:85 , srv1216745527:85 , srv1269832296:85 , srv1283958938:85 , srv130927963:85 , srv1380014758:85 , srv1401498991:85 , srv1444684264:85 , srv1476954640:85 , srv1485681200:85 , srv1544494490:85 , srv1545256365:85 , srv1588577461:85 , srv1643098520:85 , srv1734446232:85 , srv1748352628:85 , srv1754152942:85 , srv1761415484:85 , srv1822949218:85 , srv1837205862:85 , srv1934616135:85 , srv2121911234:85 , srv2146562111:85 , srv298191604:85 , srv319145232:85 , srv373833666:85 , srv449696393:85 , srv548066447:85 , srv583518834:85 , srv623261566:85 , srv624262984:85 , srv634663475:85 , srv654533434:85 , srv680805238:85 , srv694491579:85 , srv710404478:85 , srv720449578:85 , srv722937079:85 , srv725006819:85 , srv730964212:85 , srv807311457:85 , srv831426590:85 , srv841038898:85 , srv84908664:85 , srv864099257:85 , srv865406278:85 , srv873534218:85 , srv875306744:85 , srv930238814:85 , srv998760019:85 , srv1145582048:86 , srv1189823533:86 , srv1317837699:86 , srv1367807547:86 , srv1457854849:86 , srv1483193548:86 , srv1854114441:86 , srv2016925776:86 , srv2074961396:86 , srv522925931:86 , srv876929712:86 , srv880130260:86 , srv219763387:87 , srv1061397530:614 } 2023-02-11 19:56:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:56:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373833666=65, srv728694941=83, srv1380014758=22, srv1646042753=38, srv880130260=95, srv1146915614=7, srv150730910=31, srv875306744=93, srv354130561=64, srv1476954640=27, srv1653092175=39, srv1364533382=20, srv2016925776=53, srv2121911234=56, srv1214233338=11, srv219763387=59, srv599894188=72, srv710404478=79, srv1045669613=2, srv654533434=76, srv1539323434=33, srv1185871978=9, srv298191604=61, srv1545256365=35, srv522925931=68, srv1290155068=16, srv1837205862=47, srv720449578=80, srv930238814=97, srv399584438=66, srv1283958938=15, srv1544494490=34, srv1367807547=21, srv831426590=87, srv928727869=96, srv1444684264=25, srv449696393=67, srv680805238=77, srv1075125727=5, srv757296558=85, srv864099257=90, srv1748352628=42, srv1761415484=44, srv876929712=94, srv1033033887=1, srv309244707=62, srv841038898=88, srv1588577461=36, srv2146562111=57, srv1358370848=19, srv1502761281=30, srv1189823533=10, srv1317837699=18, srv865406278=91, srv722937079=81, srv1934616135=51, srv1840561370=48, srv998760019=99, srv1485681200=29, srv1822949218=46, srv319145232=63, srv1266784275=13, srv1269832296=14, srv1216745527=12, srv624262984=74, srv634663475=75, srv807311457=86, srv534409875=69, srv1643098520=37, srv84908664=89, srv1754152942=43, srv1403871113=24, srv190746759=50, srv1958700076=52, srv1795548121=45, srv873534218=92, srv694491579=78, srv130927963=17, srv583518834=71, srv548066447=70, srv284501302=60, srv1176457946=8, srv623261566=73, srv1538548986=32, srv1401498991=23, srv1009593933=0, srv1675609273=40, srv214681506=58, srv1061397530=4, srv1854114441=49, srv2074961396=55, srv1483193548=28, srv1734446232=41, srv1050849155=3, srv2060890343=54, srv1145582048=6, srv725006819=82, srv730964212=84, srv941293933=98, srv1457854849=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:56:00,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:56:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-11 19:56:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-11 19:56:00,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-11 19:56:00,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-11 19:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-11 19:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-11 19:56:01,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:01,066 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-11 19:56:01,066 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.6710768554601124E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-11 19:56:01,281 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 309 ms to try 5000 different iterations. Found a solution that moves 54 regions; Going from a computed imbalance of 3.6710768554601124E-5 to a new imbalance of 6.776101753906762E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.006); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:01,282 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1075125727:83 , srv1367807547:83 , srv1544494490:83 , srv1734446232:83 , srv1837205862:83 , srv1854114441:83 , srv2016925776:83 , srv219763387:83 , srv725006819:83 , srv864099257:83 , srv930238814:83 , srv1033033887:84 , srv1045669613:84 , srv1145582048:84 , srv1146915614:84 , srv1176457946:84 , srv1185871978:84 , srv1189823533:84 , srv1214233338:84 , srv1216745527:84 , srv1266784275:84 , srv1269832296:84 , srv1283958938:84 , srv130927963:84 , srv1317837699:84 , srv1358370848:84 , srv1380014758:84 , srv1401498991:84 , srv1403871113:84 , srv1444684264:84 , srv1457854849:84 , srv1476954640:84 , srv1483193548:84 , srv1485681200:84 , srv1502761281:84 , srv1539323434:84 , srv1545256365:84 , srv1588577461:84 , srv1643098520:84 , srv1646042753:84 , srv1653092175:84 , srv1748352628:84 , srv1754152942:84 , srv1761415484:84 , srv1795548121:84 , srv1822949218:84 , srv1934616135:84 , srv1958700076:84 , srv2060890343:84 , srv2074961396:84 , srv2121911234:84 , srv2146562111:84 , srv214681506:84 , srv298191604:84 , srv309244707:84 , srv319145232:84 , srv354130561:84 , srv373833666:84 , srv399584438:84 , srv449696393:84 , srv522925931:84 , srv534409875:84 , srv548066447:84 , srv583518834:84 , srv623261566:84 , srv624262984:84 , srv634663475:84 , srv654533434:84 , srv680805238:84 , srv694491579:84 , srv710404478:84 , srv720449578:84 , srv722937079:84 , srv728694941:84 , srv730964212:84 , srv757296558:84 , srv807311457:84 , srv831426590:84 , srv841038898:84 , srv84908664:84 , srv865406278:84 , srv873534218:84 , srv875306744:84 , srv876929712:84 , srv880130260:84 , srv928727869:84 , srv941293933:84 , srv998760019:84 , srv1009593933:85 , srv1050849155:85 , srv1290155068:85 , srv1364533382:85 , srv150730910:85 , srv1538548986:85 , srv1675609273:85 , srv1840561370:85 , srv190746759:85 , srv284501302:85 , srv599894188:85 , srv1061397530:684 } 2023-02-11 19:56:01,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:56:01,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373833666=65, srv728694941=83, srv1380014758=22, srv1646042753=38, srv880130260=95, srv1146915614=7, srv150730910=31, srv875306744=93, srv354130561=64, srv1476954640=27, srv1653092175=39, srv1364533382=20, srv2016925776=53, srv2121911234=56, srv1214233338=11, srv219763387=59, srv599894188=72, srv710404478=79, srv1045669613=2, srv654533434=76, srv1539323434=33, srv1185871978=9, srv298191604=61, srv1545256365=35, srv522925931=68, srv1290155068=16, srv1837205862=47, srv720449578=80, srv930238814=97, srv399584438=66, srv1283958938=15, srv1544494490=34, srv1367807547=21, srv831426590=87, srv928727869=96, srv1444684264=25, srv449696393=67, srv680805238=77, srv1075125727=5, srv757296558=85, srv864099257=90, srv1748352628=42, srv1761415484=44, srv876929712=94, srv1033033887=1, srv309244707=62, srv841038898=88, srv1588577461=36, srv2146562111=57, srv1358370848=19, srv1502761281=30, srv1189823533=10, srv1317837699=18, srv865406278=91, srv722937079=81, srv1934616135=51, srv1840561370=48, srv998760019=99, srv1485681200=29, srv1822949218=46, srv319145232=63, srv1266784275=13, srv1269832296=14, srv1216745527=12, srv624262984=74, srv634663475=75, srv807311457=86, srv534409875=69, srv1643098520=37, srv84908664=89, srv1754152942=43, srv1403871113=24, srv190746759=50, srv1958700076=52, srv1795548121=45, srv873534218=92, srv694491579=78, srv130927963=17, srv583518834=71, srv548066447=70, srv284501302=60, srv1176457946=8, srv623261566=73, srv1538548986=32, srv1401498991=23, srv1009593933=0, srv1675609273=40, srv214681506=58, srv1061397530=4, srv1854114441=49, srv2074961396=55, srv1483193548=28, srv1734446232=41, srv1050849155=3, srv2060890343=54, srv1145582048=6, srv725006819=82, srv730964212=84, srv941293933=98, srv1457854849=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-11 19:56:01,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-11 19:56:01,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-11 19:56:01,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-11 19:56:01,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-11 19:56:01,408 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:01,408 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1075125727:83 , srv1367807547:83 , srv1544494490:83 , srv1734446232:83 , srv1837205862:83 , srv1854114441:83 , srv2016925776:83 , srv219763387:83 , srv725006819:83 , srv864099257:83 , srv930238814:83 , srv1033033887:84 , srv1045669613:84 , srv1145582048:84 , srv1146915614:84 , srv1176457946:84 , srv1185871978:84 , srv1189823533:84 , srv1214233338:84 , srv1216745527:84 , srv1266784275:84 , srv1269832296:84 , srv1283958938:84 , srv130927963:84 , srv1317837699:84 , srv1358370848:84 , srv1380014758:84 , srv1401498991:84 , srv1403871113:84 , srv1444684264:84 , srv1457854849:84 , srv1476954640:84 , srv1483193548:84 , srv1485681200:84 , srv1502761281:84 , srv1539323434:84 , srv1545256365:84 , srv1588577461:84 , srv1643098520:84 , srv1646042753:84 , srv1653092175:84 , srv1748352628:84 , srv1754152942:84 , srv1761415484:84 , srv1795548121:84 , srv1822949218:84 , srv1934616135:84 , srv1958700076:84 , srv2060890343:84 , srv2074961396:84 , srv2121911234:84 , srv2146562111:84 , srv214681506:84 , srv298191604:84 , srv309244707:84 , srv319145232:84 , srv354130561:84 , srv373833666:84 , srv399584438:84 , srv449696393:84 , srv522925931:84 , srv534409875:84 , srv548066447:84 , srv583518834:84 , srv623261566:84 , srv624262984:84 , srv634663475:84 , srv654533434:84 , srv680805238:84 , srv694491579:84 , srv710404478:84 , srv720449578:84 , srv722937079:84 , srv728694941:84 , srv730964212:84 , srv757296558:84 , srv807311457:84 , srv831426590:84 , srv841038898:84 , srv84908664:84 , srv865406278:84 , srv873534218:84 , srv875306744:84 , srv876929712:84 , srv880130260:84 , srv928727869:84 , srv941293933:84 , srv998760019:84 , srv1009593933:85 , srv1050849155:85 , srv1290155068:85 , srv1364533382:85 , srv150730910:85 , srv1538548986:85 , srv1675609273:85 , srv1840561370:85 , srv190746759:85 , srv284501302:85 , srv599894188:85 , srv1061397530:684 } 2023-02-11 19:56:01,432 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 209), ProcessCount=168 (was 168), AvailableMemoryMB=3722 (was 4213) 2023-02-11 19:56:01,439 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=209, ProcessCount=168, AvailableMemoryMB=3722 2023-02-11 19:56:01,440 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-11 19:56:01,440 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-11 19:56:01,451 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000880092:84 , srv1209882334:84 , srv1244371352:84 , srv1400476346:84 , srv1626753561:84 , srv1642145031:84 , srv1695168943:84 , srv1703536467:84 , srv1713594018:84 , srv1784555138:84 , srv1809239438:84 , srv1820636692:84 , srv1949826836:84 , srv1959280549:84 , srv240337905:84 , srv243762680:84 , srv310087634:84 , srv333038900:84 , srv403112933:84 , srv464575642:84 , srv528032872:84 , srv53665115:84 , srv615142152:84 , srv665778388:84 , srv741342854:84 , srv741696745:84 , srv780824347:84 , srv831817379:84 , srv92749819:84 , srv768447516:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-11 19:56:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:56:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv741696745=25, srv1695168943=6, srv403112933=18, srv1209882334=1, srv615142152=22, srv1809239438=10, srv1000880092=0, srv1400476346=3, srv1713594018=8, srv1244371352=2, srv1642145031=5, srv1626753561=4, srv1949826836=12, srv243762680=15, srv528032872=20, srv780824347=27, srv92749819=29, srv1959280549=13, srv741342854=24, srv1784555138=9, srv768447516=26, srv464575642=19, srv240337905=14, srv665778388=23, srv333038900=17, srv1703536467=7, srv1820636692=11, srv831817379=28, srv53665115=21, srv310087634=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:56:01,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:56:01,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:56:01,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:56:01,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:56:01,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:56:01,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:56:01,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:56:01,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-11 19:56:01,494 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1533048456798503, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:56:01,494 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912676101827153, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1533048456798503, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-11 19:58:01,473 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6182420 different iterations. Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912676101827153 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:58:01,478 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1000880092:90 , srv1209882334:90 , srv1244371352:90 , srv1400476346:90 , srv1626753561:90 , srv1642145031:90 , srv1695168943:90 , srv1703536467:90 , srv1713594018:90 , srv1784555138:90 , srv1809239438:90 , srv1820636692:90 , srv1949826836:90 , srv1959280549:90 , srv240337905:90 , srv243762680:90 , srv310087634:90 , srv333038900:90 , srv403112933:90 , srv464575642:90 , srv528032872:90 , srv53665115:90 , srv615142152:90 , srv665778388:90 , srv741342854:90 , srv741696745:90 , srv768447516:90 , srv780824347:90 , srv831817379:90 , srv92749819:90 } 2023-02-11 19:58:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-11 19:58:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv741696745=25, srv1695168943=6, srv403112933=18, srv1209882334=1, srv615142152=22, srv1809239438=10, srv1000880092=0, srv1400476346=3, srv1713594018=8, srv1244371352=2, srv1642145031=5, srv1626753561=4, srv1949826836=12, srv243762680=15, srv528032872=20, srv780824347=27, srv92749819=29, srv1959280549=13, srv741342854=24, srv1784555138=9, srv768447516=26, srv464575642=19, srv240337905=14, srv665778388=23, srv333038900=17, srv1703536467=7, srv1820636692=11, srv831817379=28, srv53665115=21, srv310087634=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-11 19:58:01,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-11 19:58:01,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-11 19:58:01,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-11 19:58:01,499 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-11 19:58:01,499 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1000880092:90 , srv1209882334:90 , srv1244371352:90 , srv1400476346:90 , srv1626753561:90 , srv1642145031:90 , srv1695168943:90 , srv1703536467:90 , srv1713594018:90 , srv1784555138:90 , srv1809239438:90 , srv1820636692:90 , srv1949826836:90 , srv1959280549:90 , srv240337905:90 , srv243762680:90 , srv310087634:90 , srv333038900:90 , srv403112933:90 , srv464575642:90 , srv528032872:90 , srv53665115:90 , srv615142152:90 , srv665778388:90 , srv741342854:90 , srv741696745:90 , srv768447516:90 , srv780824347:90 , srv831817379:90 , srv92749819:90 } 2023-02-11 19:58:01,511 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=256 (was 209) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 168), AvailableMemoryMB=3315 (was 3722)