2023-06-08 19:01:23,665 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-06-08 19:01:23,897 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-06-08 19:01:24,002 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-06-08 19:01:24,053 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-08 19:01:24,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-08 19:01:24,073 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=181, ProcessCount=183, AvailableMemoryMB=1760 2023-06-08 19:01:24,074 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-08 19:01:24,075 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-08 19:01:24,181 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000996154:84 , srv1018830567:84 , srv1036128985:84 , srv1055331824:84 , srv1085123693:84 , srv1106474857:84 , srv115373444:84 , srv1177871963:84 , srv122791043:84 , srv124622969:84 , srv1252355071:84 , srv1262830624:84 , srv1273933892:84 , srv1285976741:84 , srv1352735401:84 , srv1359147799:84 , srv136641061:84 , srv1368606473:84 , srv1387439721:84 , srv1399013760:84 , srv1407962106:84 , srv1442407448:84 , srv1444156988:84 , srv1466693690:84 , srv1467810369:84 , srv1487422:84 , srv1491633588:84 , srv1511483623:84 , srv1514636577:84 , srv1534800627:84 , srv1546118541:84 , srv1550593698:84 , srv1566103305:84 , srv1573215630:84 , srv1623575241:84 , srv1644059286:84 , srv1651843319:84 , srv1698196124:84 , srv1716817457:84 , srv173416321:84 , srv1757288451:84 , srv1765950583:84 , srv1766635286:84 , srv1801965037:84 , srv1803585993:84 , srv188813003:84 , srv1914322346:84 , srv1963878624:84 , srv1990147313:84 , srv2004064776:84 , srv2024881847:84 , srv2059060499:84 , srv2060857890:84 , srv2061468326:84 , srv2092092846:84 , srv2120548291:84 , srv217402335:84 , srv267512922:84 , srv270146612:84 , srv307391110:84 , srv31097278:84 , srv330799414:84 , srv346898844:84 , srv348001650:84 , srv379894184:84 , srv40320293:84 , srv42789815:84 , srv478839780:84 , srv479634137:84 , srv484597035:84 , srv495517474:84 , srv517621298:84 , srv555525965:84 , srv566582974:84 , srv579884609:84 , srv58312656:84 , srv586369811:84 , srv589440923:84 , srv609153991:84 , srv622073994:84 , srv680577122:84 , srv704988355:84 , srv70544593:84 , srv707270278:84 , srv71253856:84 , srv714932915:84 , srv758469449:84 , srv76317581:84 , srv803826843:84 , srv810899018:84 , srv850218572:84 , srv866736051:84 , srv880417610:84 , srv926628572:84 , srv936179363:84 , srv954548864:84 , srv966026864:84 , srv970887884:84 , srv979546037:84 , srv927333060:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-06-08 19:01:24,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:24,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555525965=72, srv1491633588=26, srv136641061=16, srv1550593698=31, srv1018830567=1, srv622073994=79, srv1623575241=34, srv758469449=86, srv1407962106=20, srv71253856=84, srv1566103305=32, srv1803585993=44, srv2061468326=53, srv1963878624=47, srv495517474=70, srv1252355071=10, srv188813003=45, srv346898844=62, srv936179363=95, srv1106474857=5, srv1359147799=15, srv2024881847=50, srv2060857890=52, srv1765950583=41, srv2059060499=51, srv40320293=65, srv566582974=73, srv1399013760=19, srv589440923=77, srv880417610=92, srv1000996154=0, srv1534800627=29, srv704988355=81, srv1573215630=33, srv217402335=56, srv270146612=58, srv479634137=68, srv1511483623=27, srv1716817457=38, srv2092092846=54, srv31097278=60, srv58312656=75, srv1085123693=4, srv1990147313=48, srv330799414=61, srv707270278=83, srv42789815=66, srv70544593=82, srv586369811=76, srv803826843=88, srv966026864=97, srv1651843319=36, srv267512922=57, srv1442407448=21, srv1801965037=43, srv810899018=89, srv1644059286=35, srv1444156988=22, srv1467810369=24, srv609153991=78, srv579884609=74, srv1387439721=18, srv2120548291=55, srv1055331824=3, srv122791043=8, srv979546037=99, srv76317581=87, srv850218572=90, srv1177871963=7, srv1352735401=14, srv1698196124=37, srv307391110=59, srv484597035=69, srv2004064776=49, srv1285976741=13, srv124622969=9, srv379894184=64, srv517621298=71, srv1273933892=12, srv1368606473=17, srv680577122=80, srv115373444=6, srv927333060=94, srv1487422=25, srv1914322346=46, srv478839780=67, srv866736051=91, srv954548864=96, srv1514636577=28, srv970887884=98, srv1466693690=23, srv1546118541=30, srv1757288451=40, srv1262830624=11, srv173416321=39, srv1036128985=2, srv714932915=85, srv1766635286=42, srv926628572=93, srv348001650=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:24,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:24,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:24,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:24,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:24,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:24,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:24,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:24,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 19:01:24,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 19:01:24,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 19:01:24,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 19:01:24,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 19:01:24,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 19:01:24,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 19:01:24,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 19:01:24,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 19:01:24,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:24,537 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 19:01:24,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 19:01:24,925 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 633 ms to try 5000 different iterations. Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17763896844799332. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03794279664750076, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05196260418267705, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03794279664750076, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17291666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.239, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:24,934 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1407962106:81 , srv1487422:81 , srv1698196124:81 , srv579884609:81 , srv2060857890:82 , srv1177871963:83 , srv1566103305:83 , srv1573215630:83 , srv188813003:83 , srv1990147313:83 , srv479634137:83 , srv866736051:83 , srv1018830567:84 , srv1801965037:84 , srv1914322346:84 , srv2120548291:84 , srv217402335:84 , srv267512922:84 , srv31097278:84 , srv40320293:84 , srv42789815:84 , srv566582974:84 , srv680577122:84 , srv704988355:84 , srv124622969:85 , srv1262830624:85 , srv1352735401:85 , srv136641061:85 , srv1444156988:85 , srv1466693690:85 , srv586369811:85 , srv71253856:85 , srv758469449:85 , srv803826843:85 , srv880417610:85 , srv115373444:86 , srv1273933892:86 , srv1368606473:86 , srv1803585993:86 , srv2092092846:86 , srv270146612:86 , srv346898844:86 , srv478839780:86 , srv517621298:86 , srv707270278:86 , srv76317581:86 , srv966026864:86 , srv970887884:86 , srv1106474857:87 , srv1285976741:87 , srv1359147799:87 , srv1387439721:87 , srv1399013760:87 , srv1491633588:87 , srv1511483623:87 , srv1514636577:87 , srv1550593698:87 , srv2004064776:87 , srv2061468326:87 , srv330799414:87 , srv484597035:87 , srv58312656:87 , srv609153991:87 , srv70544593:87 , srv1036128985:88 , srv1252355071:88 , srv1467810369:88 , srv1765950583:88 , srv2024881847:88 , srv307391110:88 , srv348001650:88 , srv379894184:88 , srv495517474:88 , srv622073994:88 , srv810899018:88 , srv954548864:88 , srv1000996154:89 , srv1442407448:89 , srv1534800627:89 , srv1623575241:89 , srv1716817457:89 , srv555525965:89 , srv979546037:89 , srv1055331824:90 , srv1085123693:90 , srv122791043:90 , srv1644059286:90 , srv1766635286:90 , srv1963878624:90 , srv589440923:90 , srv714932915:90 , srv1546118541:91 , srv1757288451:91 , srv926628572:91 , srv936179363:91 , srv1651843319:92 , srv2059060499:92 , srv850218572:92 , srv173416321:94 , srv927333060:427 } 2023-06-08 19:01:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555525965=72, srv1491633588=26, srv136641061=16, srv1550593698=31, srv1018830567=1, srv622073994=79, srv1623575241=34, srv758469449=86, srv1407962106=20, srv71253856=84, srv1566103305=32, srv1803585993=44, srv2061468326=53, srv1963878624=47, srv495517474=70, srv1252355071=10, srv188813003=45, srv346898844=62, srv936179363=95, srv1106474857=5, srv1359147799=15, srv2024881847=50, srv2060857890=52, srv1765950583=41, srv2059060499=51, srv40320293=65, srv566582974=73, srv1399013760=19, srv589440923=77, srv880417610=92, srv1000996154=0, srv1534800627=29, srv704988355=81, srv1573215630=33, srv217402335=56, srv270146612=58, srv479634137=68, srv1511483623=27, srv1716817457=38, srv2092092846=54, srv31097278=60, srv58312656=75, srv1085123693=4, srv1990147313=48, srv330799414=61, srv707270278=83, srv42789815=66, srv70544593=82, srv586369811=76, srv803826843=88, srv966026864=97, srv1651843319=36, srv267512922=57, srv1442407448=21, srv1801965037=43, srv810899018=89, srv1644059286=35, srv1444156988=22, srv1467810369=24, srv609153991=78, srv579884609=74, srv1387439721=18, srv2120548291=55, srv1055331824=3, srv122791043=8, srv979546037=99, srv76317581=87, srv850218572=90, srv1177871963=7, srv1352735401=14, srv1698196124=37, srv307391110=59, srv484597035=69, srv2004064776=49, srv1285976741=13, srv124622969=9, srv379894184=64, srv517621298=71, srv1273933892=12, srv1368606473=17, srv680577122=80, srv115373444=6, srv927333060=94, srv1487422=25, srv1914322346=46, srv478839780=67, srv866736051=91, srv954548864=96, srv1514636577=28, srv970887884=98, srv1466693690=23, srv1546118541=30, srv1757288451=40, srv1262830624=11, srv173416321=39, srv1036128985=2, srv714932915=85, srv1766635286=42, srv926628572=93, srv348001650=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 19:01:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 19:01:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 19:01:24,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 19:01:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 19:01:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 19:01:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 19:01:25,108 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03794279664750076, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05196260418267705, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03794279664750076, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17291666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.239, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:25,108 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 19:01:25,109 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17760605134244192, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03794279664750076, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05196260418267705, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03794279664750076, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17291666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.239, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 19:01:25,313 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 338 ms to try 5000 different iterations. Found a solution that moves 2642 regions; Going from a computed imbalance of 0.17760605134244192 to a new imbalance of 0.014420964726763203. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021960864873298812); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03440042805047804, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29355555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021960864873298812); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:25,325 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv173416321:79 , srv926628572:79 , srv1000996154:80 , srv1766635286:80 , srv2059060499:80 , srv1085123693:81 , srv1644059286:81 , srv1757288451:81 , srv1442407448:82 , srv1651843319:82 , srv1765950583:82 , srv714932915:82 , srv850218572:82 , srv936179363:82 , srv122791043:83 , srv1399013760:83 , srv1467810369:83 , srv1514636577:83 , srv1546118541:83 , srv1550593698:83 , srv1623575241:83 , srv1716817457:83 , srv2061468326:83 , srv979546037:83 , srv1036128985:84 , srv115373444:84 , srv1387439721:84 , srv1534800627:84 , srv2024881847:84 , srv31097278:84 , srv348001650:84 , srv379894184:84 , srv495517474:84 , srv555525965:84 , srv589440923:84 , srv609153991:84 , srv622073994:84 , srv810899018:84 , srv954548864:84 , srv1106474857:85 , srv1252355071:85 , srv1491633588:85 , srv1511483623:85 , srv1803585993:85 , srv1963878624:85 , srv267512922:85 , srv270146612:85 , srv586369811:85 , srv707270278:85 , srv803826843:85 , srv1055331824:86 , srv1262830624:86 , srv1566103305:86 , srv1801965037:86 , srv1914322346:86 , srv307391110:86 , srv478839780:86 , srv484597035:86 , srv517621298:86 , srv58312656:86 , srv76317581:86 , srv880417610:86 , srv966026864:86 , srv970887884:86 , srv1018830567:87 , srv1273933892:87 , srv1359147799:87 , srv1368606473:87 , srv188813003:87 , srv2004064776:87 , srv2060857890:87 , srv2120548291:87 , srv330799414:87 , srv346898844:87 , srv42789815:87 , srv680577122:87 , srv71253856:87 , srv866736051:87 , srv1285976741:88 , srv1352735401:88 , srv1466693690:88 , srv2092092846:88 , srv758469449:88 , srv124622969:89 , srv136641061:89 , srv1990147313:89 , srv566582974:89 , srv70544593:89 , srv1407962106:90 , srv1444156988:90 , srv1487422:90 , srv1573215630:90 , srv217402335:90 , srv40320293:90 , srv579884609:90 , srv704988355:90 , srv1177871963:91 , srv1698196124:91 , srv479634137:91 , srv927333060:542 } 2023-06-08 19:01:25,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:25,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555525965=72, srv1491633588=26, srv136641061=16, srv1550593698=31, srv1018830567=1, srv622073994=79, srv1623575241=34, srv758469449=86, srv1407962106=20, srv71253856=84, srv1566103305=32, srv1803585993=44, srv2061468326=53, srv1963878624=47, srv495517474=70, srv1252355071=10, srv188813003=45, srv346898844=62, srv936179363=95, srv1106474857=5, srv1359147799=15, srv2024881847=50, srv2060857890=52, srv1765950583=41, srv2059060499=51, srv40320293=65, srv566582974=73, srv1399013760=19, srv589440923=77, srv880417610=92, srv1000996154=0, srv1534800627=29, srv704988355=81, srv1573215630=33, srv217402335=56, srv270146612=58, srv479634137=68, srv1511483623=27, srv1716817457=38, srv2092092846=54, srv31097278=60, srv58312656=75, srv1085123693=4, srv1990147313=48, srv330799414=61, srv707270278=83, srv42789815=66, srv70544593=82, srv586369811=76, srv803826843=88, srv966026864=97, srv1651843319=36, srv267512922=57, srv1442407448=21, srv1801965037=43, srv810899018=89, srv1644059286=35, srv1444156988=22, srv1467810369=24, srv609153991=78, srv579884609=74, srv1387439721=18, srv2120548291=55, srv1055331824=3, srv122791043=8, srv979546037=99, srv76317581=87, srv850218572=90, srv1177871963=7, srv1352735401=14, srv1698196124=37, srv307391110=59, srv484597035=69, srv2004064776=49, srv1285976741=13, srv124622969=9, srv379894184=64, srv517621298=71, srv1273933892=12, srv1368606473=17, srv680577122=80, srv115373444=6, srv927333060=94, srv1487422=25, srv1914322346=46, srv478839780=67, srv866736051=91, srv954548864=96, srv1514636577=28, srv970887884=98, srv1466693690=23, srv1546118541=30, srv1757288451=40, srv1262830624=11, srv173416321=39, srv1036128985=2, srv714932915=85, srv1766635286=42, srv926628572=93, srv348001650=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:25,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:25,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 19:01:25,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 19:01:25,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 19:01:25,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 19:01:25,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 19:01:25,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 19:01:25,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 19:01:25,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 19:01:25,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 19:01:25,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 19:01:25,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 19:01:25,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 19:01:25,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 19:01:25,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 19:01:25,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021960864873298812); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03440042805047804, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021960864873298812); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:25,451 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 19:01:25,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014402465047361626, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021960864873298812); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03440042805047804, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021960864873298812); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 19:01:25,648 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 274 ms to try 5000 different iterations. Found a solution that moves 956 regions; Going from a computed imbalance of 0.014402465047361626 to a new imbalance of 6.353597293769897E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005645732772720048); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008619660142094213); MoveCostFunction : (multiplier=7.0, imbalance=0.10622222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005645732772720048); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:25,652 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv70544593:81 , srv1055331824:82 , srv1285976741:82 , srv124622969:83 , srv1359147799:83 , srv1444156988:83 , srv1963878624:83 , srv2004064776:83 , srv2092092846:83 , srv330799414:83 , srv704988355:83 , srv1177871963:84 , srv122791043:84 , srv1252355071:84 , srv1352735401:84 , srv136641061:84 , srv1466693690:84 , srv1546118541:84 , srv1573215630:84 , srv1623575241:84 , srv217402335:84 , srv307391110:84 , srv346898844:84 , srv40320293:84 , srv479634137:84 , srv589440923:84 , srv758469449:84 , srv850218572:84 , srv936179363:84 , srv1036128985:85 , srv1273933892:85 , srv1368606473:85 , srv1534800627:85 , srv1651843319:85 , srv1698196124:85 , srv1716817457:85 , srv173416321:85 , srv2024881847:85 , srv484597035:85 , srv495517474:85 , srv517621298:85 , srv555525965:85 , srv566582974:85 , srv58312656:85 , srv622073994:85 , srv714932915:85 , srv970887884:85 , srv1085123693:86 , srv1106474857:86 , srv1262830624:86 , srv1407962106:86 , srv1467810369:86 , srv1487422:86 , srv1491633588:86 , srv1511483623:86 , srv1644059286:86 , srv1757288451:86 , srv1803585993:86 , srv1990147313:86 , srv2059060499:86 , srv2120548291:86 , srv270146612:86 , srv348001650:86 , srv379894184:86 , srv579884609:86 , srv609153991:86 , srv680577122:86 , srv71253856:86 , srv76317581:86 , srv810899018:86 , srv880417610:86 , srv954548864:86 , srv966026864:86 , srv979546037:86 , srv1018830567:87 , srv115373444:87 , srv1387439721:87 , srv1399013760:87 , srv1442407448:87 , srv1766635286:87 , srv1801965037:87 , srv1914322346:87 , srv42789815:87 , srv478839780:87 , srv586369811:87 , srv707270278:87 , srv866736051:87 , srv926628572:87 , srv1000996154:88 , srv1514636577:88 , srv1550593698:88 , srv1566103305:88 , srv1765950583:88 , srv188813003:88 , srv2061468326:88 , srv267512922:88 , srv803826843:88 , srv2060857890:89 , srv31097278:89 , srv927333060:539 } 2023-06-08 19:01:25,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:25,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555525965=72, srv1491633588=26, srv136641061=16, srv1550593698=31, srv1018830567=1, srv622073994=79, srv1623575241=34, srv758469449=86, srv1407962106=20, srv71253856=84, srv1566103305=32, srv1803585993=44, srv2061468326=53, srv1963878624=47, srv495517474=70, srv1252355071=10, srv188813003=45, srv346898844=62, srv936179363=95, srv1106474857=5, srv1359147799=15, srv2024881847=50, srv2060857890=52, srv1765950583=41, srv2059060499=51, srv40320293=65, srv566582974=73, srv1399013760=19, srv589440923=77, srv880417610=92, srv1000996154=0, srv1534800627=29, srv704988355=81, srv1573215630=33, srv217402335=56, srv270146612=58, srv479634137=68, srv1511483623=27, srv1716817457=38, srv2092092846=54, srv31097278=60, srv58312656=75, srv1085123693=4, srv1990147313=48, srv330799414=61, srv707270278=83, srv42789815=66, srv70544593=82, srv586369811=76, srv803826843=88, srv966026864=97, srv1651843319=36, srv267512922=57, srv1442407448=21, srv1801965037=43, srv810899018=89, srv1644059286=35, srv1444156988=22, srv1467810369=24, srv609153991=78, srv579884609=74, srv1387439721=18, srv2120548291=55, srv1055331824=3, srv122791043=8, srv979546037=99, srv76317581=87, srv850218572=90, srv1177871963=7, srv1352735401=14, srv1698196124=37, srv307391110=59, srv484597035=69, srv2004064776=49, srv1285976741=13, srv124622969=9, srv379894184=64, srv517621298=71, srv1273933892=12, srv1368606473=17, srv680577122=80, srv115373444=6, srv927333060=94, srv1487422=25, srv1914322346=46, srv478839780=67, srv866736051=91, srv954548864=96, srv1514636577=28, srv970887884=98, srv1466693690=23, srv1546118541=30, srv1757288451=40, srv1262830624=11, srv173416321=39, srv1036128985=2, srv714932915=85, srv1766635286=42, srv926628572=93, srv348001650=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:25,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:25,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:25,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:25,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:25,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:25,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:25,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 19:01:25,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 19:01:25,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 19:01:25,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 19:01:25,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 19:01:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 19:01:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 19:01:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 19:01:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 19:01:25,779 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005645732772720048); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008619660142094213); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005645732772720048); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:25,780 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 19:01:25,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.286656743020818E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005645732772720048); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008619660142094213); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005645732772720048); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 19:01:26,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 320 ms to try 5000 different iterations. Found a solution that moves 259 regions; Going from a computed imbalance of 6.286656743020818E-4 to a new imbalance of 3.6462613520805696E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.028777777777777777, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:26,019 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1651843319:83 , srv42789815:83 , srv478839780:83 , srv850218572:83 , srv1018830567:84 , srv1055331824:84 , srv1177871963:84 , srv1252355071:84 , srv1273933892:84 , srv1285976741:84 , srv1352735401:84 , srv136641061:84 , srv1368606473:84 , srv1387439721:84 , srv1399013760:84 , srv1442407448:84 , srv1444156988:84 , srv1491633588:84 , srv1511483623:84 , srv1514636577:84 , srv1546118541:84 , srv1550593698:84 , srv1716817457:84 , srv173416321:84 , srv1757288451:84 , srv1765950583:84 , srv188813003:84 , srv1963878624:84 , srv1990147313:84 , srv2059060499:84 , srv2060857890:84 , srv2061468326:84 , srv2092092846:84 , srv217402335:84 , srv307391110:84 , srv330799414:84 , srv346898844:84 , srv348001650:84 , srv379894184:84 , srv479634137:84 , srv484597035:84 , srv517621298:84 , srv555525965:84 , srv58312656:84 , srv589440923:84 , srv70544593:84 , srv707270278:84 , srv803826843:84 , srv954548864:84 , srv966026864:84 , srv979546037:84 , srv1036128985:85 , srv1085123693:85 , srv1106474857:85 , srv115373444:85 , srv1262830624:85 , srv1359147799:85 , srv1407962106:85 , srv1466693690:85 , srv1467810369:85 , srv1487422:85 , srv1534800627:85 , srv1566103305:85 , srv1573215630:85 , srv1623575241:85 , srv1644059286:85 , srv1698196124:85 , srv1766635286:85 , srv1801965037:85 , srv1914322346:85 , srv2004064776:85 , srv2024881847:85 , srv267512922:85 , srv270146612:85 , srv31097278:85 , srv40320293:85 , srv495517474:85 , srv566582974:85 , srv579884609:85 , srv586369811:85 , srv609153991:85 , srv680577122:85 , srv704988355:85 , srv71253856:85 , srv714932915:85 , srv758469449:85 , srv76317581:85 , srv810899018:85 , srv866736051:85 , srv880417610:85 , srv926628572:85 , srv970887884:85 , srv1000996154:86 , srv122791043:86 , srv124622969:86 , srv1803585993:86 , srv2120548291:86 , srv622073994:86 , srv936179363:86 , srv927333060:633 } 2023-06-08 19:01:26,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:26,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555525965=72, srv1491633588=26, srv136641061=16, srv1550593698=31, srv1018830567=1, srv622073994=79, srv1623575241=34, srv758469449=86, srv1407962106=20, srv71253856=84, srv1566103305=32, srv1803585993=44, srv2061468326=53, srv1963878624=47, srv495517474=70, srv1252355071=10, srv188813003=45, srv346898844=62, srv936179363=95, srv1106474857=5, srv1359147799=15, srv2024881847=50, srv2060857890=52, srv1765950583=41, srv2059060499=51, srv40320293=65, srv566582974=73, srv1399013760=19, srv589440923=77, srv880417610=92, srv1000996154=0, srv1534800627=29, srv704988355=81, srv1573215630=33, srv217402335=56, srv270146612=58, srv479634137=68, srv1511483623=27, srv1716817457=38, srv2092092846=54, srv31097278=60, srv58312656=75, srv1085123693=4, srv1990147313=48, srv330799414=61, srv707270278=83, srv42789815=66, srv70544593=82, srv586369811=76, srv803826843=88, srv966026864=97, srv1651843319=36, srv267512922=57, srv1442407448=21, srv1801965037=43, srv810899018=89, srv1644059286=35, srv1444156988=22, srv1467810369=24, srv609153991=78, srv579884609=74, srv1387439721=18, srv2120548291=55, srv1055331824=3, srv122791043=8, srv979546037=99, srv76317581=87, srv850218572=90, srv1177871963=7, srv1352735401=14, srv1698196124=37, srv307391110=59, srv484597035=69, srv2004064776=49, srv1285976741=13, srv124622969=9, srv379894184=64, srv517621298=71, srv1273933892=12, srv1368606473=17, srv680577122=80, srv115373444=6, srv927333060=94, srv1487422=25, srv1914322346=46, srv478839780=67, srv866736051=91, srv954548864=96, srv1514636577=28, srv970887884=98, srv1466693690=23, srv1546118541=30, srv1757288451=40, srv1262830624=11, srv173416321=39, srv1036128985=2, srv714932915=85, srv1766635286=42, srv926628572=93, srv348001650=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:26,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 19:01:26,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 19:01:26,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 19:01:26,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 19:01:26,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 19:01:26,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 19:01:26,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 19:01:26,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 19:01:26,135 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:26,135 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-08 19:01:26,135 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.464905675887979E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-08 19:01:26,287 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 238 ms to try 5000 different iterations. Found a solution that moves 36 regions; Going from a computed imbalance of 3.464905675887979E-5 to a new imbalance of 6.99614644158999E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.004); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:26,288 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1000996154:83 , srv1106474857:83 , srv122791043:83 , srv124622969:83 , srv1534800627:83 , srv1803585993:83 , srv2120548291:83 , srv40320293:83 , srv566582974:83 , srv622073994:83 , srv71253856:83 , srv76317581:83 , srv810899018:83 , srv936179363:83 , srv1018830567:84 , srv1036128985:84 , srv1055331824:84 , srv1085123693:84 , srv115373444:84 , srv1177871963:84 , srv1262830624:84 , srv1273933892:84 , srv1359147799:84 , srv136641061:84 , srv1368606473:84 , srv1387439721:84 , srv1407962106:84 , srv1442407448:84 , srv1444156988:84 , srv1466693690:84 , srv1467810369:84 , srv1487422:84 , srv1491633588:84 , srv1511483623:84 , srv1514636577:84 , srv1546118541:84 , srv1550593698:84 , srv1566103305:84 , srv1573215630:84 , srv1644059286:84 , srv1651843319:84 , srv1698196124:84 , srv173416321:84 , srv1757288451:84 , srv1765950583:84 , srv1766635286:84 , srv1801965037:84 , srv188813003:84 , srv1914322346:84 , srv1963878624:84 , srv1990147313:84 , srv2004064776:84 , srv2024881847:84 , srv2059060499:84 , srv2060857890:84 , srv2061468326:84 , srv217402335:84 , srv267512922:84 , srv270146612:84 , srv307391110:84 , srv31097278:84 , srv348001650:84 , srv379894184:84 , srv478839780:84 , srv479634137:84 , srv484597035:84 , srv495517474:84 , srv555525965:84 , srv579884609:84 , srv58312656:84 , srv586369811:84 , srv589440923:84 , srv609153991:84 , srv680577122:84 , srv704988355:84 , srv707270278:84 , srv714932915:84 , srv758469449:84 , srv803826843:84 , srv866736051:84 , srv880417610:84 , srv926628572:84 , srv954548864:84 , srv966026864:84 , srv970887884:84 , srv979546037:84 , srv1252355071:85 , srv1285976741:85 , srv1352735401:85 , srv1399013760:85 , srv1623575241:85 , srv1716817457:85 , srv2092092846:85 , srv330799414:85 , srv346898844:85 , srv42789815:85 , srv517621298:85 , srv70544593:85 , srv850218572:85 , srv927333060:685 } 2023-06-08 19:01:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555525965=72, srv1491633588=26, srv136641061=16, srv1550593698=31, srv1018830567=1, srv622073994=79, srv1623575241=34, srv758469449=86, srv1407962106=20, srv71253856=84, srv1566103305=32, srv1803585993=44, srv2061468326=53, srv1963878624=47, srv495517474=70, srv1252355071=10, srv188813003=45, srv346898844=62, srv936179363=95, srv1106474857=5, srv1359147799=15, srv2024881847=50, srv2060857890=52, srv1765950583=41, srv2059060499=51, srv40320293=65, srv566582974=73, srv1399013760=19, srv589440923=77, srv880417610=92, srv1000996154=0, srv1534800627=29, srv704988355=81, srv1573215630=33, srv217402335=56, srv270146612=58, srv479634137=68, srv1511483623=27, srv1716817457=38, srv2092092846=54, srv31097278=60, srv58312656=75, srv1085123693=4, srv1990147313=48, srv330799414=61, srv707270278=83, srv42789815=66, srv70544593=82, srv586369811=76, srv803826843=88, srv966026864=97, srv1651843319=36, srv267512922=57, srv1442407448=21, srv1801965037=43, srv810899018=89, srv1644059286=35, srv1444156988=22, srv1467810369=24, srv609153991=78, srv579884609=74, srv1387439721=18, srv2120548291=55, srv1055331824=3, srv122791043=8, srv979546037=99, srv76317581=87, srv850218572=90, srv1177871963=7, srv1352735401=14, srv1698196124=37, srv307391110=59, srv484597035=69, srv2004064776=49, srv1285976741=13, srv124622969=9, srv379894184=64, srv517621298=71, srv1273933892=12, srv1368606473=17, srv680577122=80, srv115373444=6, srv927333060=94, srv1487422=25, srv1914322346=46, srv478839780=67, srv866736051=91, srv954548864=96, srv1514636577=28, srv970887884=98, srv1466693690=23, srv1546118541=30, srv1757288451=40, srv1262830624=11, srv173416321=39, srv1036128985=2, srv714932915=85, srv1766635286=42, srv926628572=93, srv348001650=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:26,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:26,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-08 19:01:26,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-08 19:01:26,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-08 19:01:26,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-08 19:01:26,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-08 19:01:26,392 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:26,393 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1000996154:83 , srv1106474857:83 , srv122791043:83 , srv124622969:83 , srv1534800627:83 , srv1803585993:83 , srv2120548291:83 , srv40320293:83 , srv566582974:83 , srv622073994:83 , srv71253856:83 , srv76317581:83 , srv810899018:83 , srv936179363:83 , srv1018830567:84 , srv1036128985:84 , srv1055331824:84 , srv1085123693:84 , srv115373444:84 , srv1177871963:84 , srv1262830624:84 , srv1273933892:84 , srv1359147799:84 , srv136641061:84 , srv1368606473:84 , srv1387439721:84 , srv1407962106:84 , srv1442407448:84 , srv1444156988:84 , srv1466693690:84 , srv1467810369:84 , srv1487422:84 , srv1491633588:84 , srv1511483623:84 , srv1514636577:84 , srv1546118541:84 , srv1550593698:84 , srv1566103305:84 , srv1573215630:84 , srv1644059286:84 , srv1651843319:84 , srv1698196124:84 , srv173416321:84 , srv1757288451:84 , srv1765950583:84 , srv1766635286:84 , srv1801965037:84 , srv188813003:84 , srv1914322346:84 , srv1963878624:84 , srv1990147313:84 , srv2004064776:84 , srv2024881847:84 , srv2059060499:84 , srv2060857890:84 , srv2061468326:84 , srv217402335:84 , srv267512922:84 , srv270146612:84 , srv307391110:84 , srv31097278:84 , srv348001650:84 , srv379894184:84 , srv478839780:84 , srv479634137:84 , srv484597035:84 , srv495517474:84 , srv555525965:84 , srv579884609:84 , srv58312656:84 , srv586369811:84 , srv589440923:84 , srv609153991:84 , srv680577122:84 , srv704988355:84 , srv707270278:84 , srv714932915:84 , srv758469449:84 , srv803826843:84 , srv866736051:84 , srv880417610:84 , srv926628572:84 , srv954548864:84 , srv966026864:84 , srv970887884:84 , srv979546037:84 , srv1252355071:85 , srv1285976741:85 , srv1352735401:85 , srv1399013760:85 , srv1623575241:85 , srv1716817457:85 , srv2092092846:85 , srv330799414:85 , srv346898844:85 , srv42789815:85 , srv517621298:85 , srv70544593:85 , srv850218572:85 , srv927333060:685 } 2023-06-08 19:01:26,415 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=190 (was 181) - SystemLoadAverage LEAK? -, ProcessCount=183 (was 183), AvailableMemoryMB=1274 (was 1760) 2023-06-08 19:01:26,423 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=190, ProcessCount=183, AvailableMemoryMB=1274 2023-06-08 19:01:26,424 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-08 19:01:26,424 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-06-08 19:01:26,431 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1063721537:84 , srv1077008925:84 , srv113797497:84 , srv1140257075:84 , srv1168708170:84 , srv1211374905:84 , srv1233619761:84 , srv1398589011:84 , srv1438523192:84 , srv1650301552:84 , srv1654950943:84 , srv1681711119:84 , srv1687479834:84 , srv1718327663:84 , srv1918188304:84 , srv1923624133:84 , srv1924368333:84 , srv1972559962:84 , srv1973715523:84 , srv1981549345:84 , srv2044918208:84 , srv227265251:84 , srv36242035:84 , srv363530429:84 , srv471866595:84 , srv598808589:84 , srv77527344:84 , srv821646870:84 , srv864579356:84 , srv1374717545:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-06-08 19:01:26,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:01:26,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1718327663=14, srv1140257075=3, srv1438523192=9, srv1972559962=18, srv598808589=26, srv77527344=27, srv36242035=23, srv363530429=24, srv1654950943=11, srv1924368333=17, srv1918188304=15, srv1681711119=12, srv1398589011=8, srv1063721537=0, srv821646870=28, srv1973715523=19, srv1687479834=13, srv471866595=25, srv2044918208=21, srv227265251=22, srv1374717545=7, srv864579356=29, srv1233619761=6, srv1211374905=5, srv1650301552=10, srv1981549345=20, srv1168708170=4, srv1923624133=16, srv1077008925=1, srv113797497=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:01:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:01:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:01:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:01:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:01:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-08 19:01:26,468 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.147833596328913, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:01:26,468 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912658862099101, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.147833596328913, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-06-08 19:03:26,450 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 8518741 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912658862099101 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:03:26,456 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1063721537:90 , srv1077008925:90 , srv113797497:90 , srv1140257075:90 , srv1168708170:90 , srv1211374905:90 , srv1233619761:90 , srv1374717545:90 , srv1398589011:90 , srv1438523192:90 , srv1650301552:90 , srv1654950943:90 , srv1681711119:90 , srv1687479834:90 , srv1718327663:90 , srv1918188304:90 , srv1923624133:90 , srv1924368333:90 , srv1972559962:90 , srv1973715523:90 , srv1981549345:90 , srv2044918208:90 , srv227265251:90 , srv36242035:90 , srv363530429:90 , srv471866595:90 , srv598808589:90 , srv77527344:90 , srv821646870:90 , srv864579356:90 } 2023-06-08 19:03:26,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-08 19:03:26,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1718327663=14, srv1140257075=3, srv1438523192=9, srv1972559962=18, srv598808589=26, srv77527344=27, srv36242035=23, srv363530429=24, srv1654950943=11, srv1924368333=17, srv1918188304=15, srv1681711119=12, srv1398589011=8, srv1063721537=0, srv821646870=28, srv1973715523=19, srv1687479834=13, srv471866595=25, srv2044918208=21, srv227265251=22, srv1374717545=7, srv864579356=29, srv1233619761=6, srv1211374905=5, srv1650301552=10, srv1981549345=20, srv1168708170=4, srv1923624133=16, srv1077008925=1, srv113797497=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-08 19:03:26,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-08 19:03:26,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-08 19:03:26,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-08 19:03:26,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-08 19:03:26,475 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1063721537:90 , srv1077008925:90 , srv113797497:90 , srv1140257075:90 , srv1168708170:90 , srv1211374905:90 , srv1233619761:90 , srv1374717545:90 , srv1398589011:90 , srv1438523192:90 , srv1650301552:90 , srv1654950943:90 , srv1681711119:90 , srv1687479834:90 , srv1718327663:90 , srv1918188304:90 , srv1923624133:90 , srv1924368333:90 , srv1972559962:90 , srv1973715523:90 , srv1981549345:90 , srv2044918208:90 , srv227265251:90 , srv36242035:90 , srv363530429:90 , srv471866595:90 , srv598808589:90 , srv77527344:90 , srv821646870:90 , srv864579356:90 } 2023-06-08 19:03:26,488 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=347 (was 190) - SystemLoadAverage LEAK? -, ProcessCount=183 (was 183), AvailableMemoryMB=1191 (was 1274)