2023-12-12 19:55:33,552 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-12 19:55:33,919 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-12 19:55:34,020 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-12 19:55:34,083 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-12 19:55:34,089 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-12 19:55:34,102 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=174, AvailableMemoryMB=4031 2023-12-12 19:55:34,103 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-12 19:55:34,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-12 19:55:34,219 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019100841:84 , srv1028640747:84 , srv1070449500:84 , srv1072264840:84 , srv1077778546:84 , srv109068049:84 , srv1093378032:84 , srv1095906840:84 , srv1100511191:84 , srv1133207750:84 , srv1149519425:84 , srv1157518354:84 , srv1190916849:84 , srv1234304578:84 , srv1273347606:84 , srv128293127:84 , srv1295264407:84 , srv1303964407:84 , srv1364600471:84 , srv140636426:84 , srv1410391633:84 , srv1459346975:84 , srv1464173198:84 , srv1478180453:84 , srv1483293538:84 , srv1497966014:84 , srv1501516420:84 , srv1519996330:84 , srv1528144278:84 , srv1568715046:84 , srv1676044950:84 , srv1690564732:84 , srv1698727444:84 , srv174313478:84 , srv1752299525:84 , srv1779048366:84 , srv1797824579:84 , srv1820663568:84 , srv1847830882:84 , srv1867076181:84 , srv1879025815:84 , srv188151044:84 , srv1890067481:84 , srv1897318501:84 , srv1898567124:84 , srv1920782173:84 , srv1935580265:84 , srv1954349585:84 , srv197518247:84 , srv1976755873:84 , srv1983324433:84 , srv2014372614:84 , srv2017443714:84 , srv2030114690:84 , srv2044969995:84 , srv2085646529:84 , srv2085880201:84 , srv2141781224:84 , srv218522795:84 , srv235306761:84 , srv255787727:84 , srv257264350:84 , srv270838116:84 , srv29235473:84 , srv294130555:84 , srv326553882:84 , srv331537872:84 , srv338480292:84 , srv356795897:84 , srv360329639:84 , srv378746305:84 , srv386706952:84 , srv39823001:84 , srv409035089:84 , srv438166455:84 , srv480137467:84 , srv482336294:84 , srv495629854:84 , srv518376412:84 , srv519594352:84 , srv580537614:84 , srv608979280:84 , srv626940684:84 , srv628808200:84 , srv633839878:84 , srv644108814:84 , srv675540463:84 , srv688958658:84 , srv766081533:84 , srv776754724:84 , srv81178558:84 , srv85359122:84 , srv863413042:84 , srv894713365:84 , srv917088438:84 , srv942973226:84 , srv958416556:84 , srv959950567:84 , srv991879183:84 , srv1369309796:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-12 19:55:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:55:34,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv675540463=87, srv1920782173=46, srv2085880201=57, srv1519996330=28, srv1879025815=41, srv29235473=64, srv438166455=75, srv1100511191=8, srv1847830882=39, srv1028640747=1, srv386706952=72, srv1364600471=18, srv1478180453=24, srv174313478=34, srv1779048366=36, srv85359122=92, srv633839878=85, srv958416556=97, srv2141781224=58, srv2085646529=56, srv1133207750=9, srv1528144278=29, srv1483293538=25, srv1752299525=35, srv1273347606=14, srv235306761=60, srv626940684=83, srv1157518354=11, srv1072264840=3, srv1410391633=21, srv294130555=65, srv766081533=89, srv1303964407=17, srv1797824579=37, srv197518247=49, srv326553882=66, srv482336294=77, srv1820663568=38, srv1077778546=4, srv917088438=95, srv1019100841=0, srv1890067481=43, srv270838116=63, srv991879183=99, srv255787727=61, srv1676044950=31, srv331537872=67, srv1983324433=51, srv409035089=74, srv1935580265=47, srv39823001=73, srv518376412=79, srv356795897=69, srv580537614=81, srv688958658=88, srv959950567=98, srv257264350=62, srv1070449500=2, srv1501516420=27, srv1898567124=45, srv1149519425=10, srv338480292=68, srv776754724=90, srv1698727444=33, srv2044969995=55, srv644108814=86, srv2014372614=52, srv1867076181=40, srv378746305=71, srv1497966014=26, srv2030114690=54, srv1954349585=48, srv128293127=15, srv81178558=91, srv1295264407=16, srv495629854=78, srv628808200=84, srv1093378032=6, srv1690564732=32, srv1568715046=30, srv1234304578=13, srv188151044=42, srv480137467=76, srv1897318501=44, srv1095906840=7, srv360329639=70, srv140636426=20, srv218522795=59, srv1190916849=12, srv1976755873=50, srv109068049=5, srv1464173198=23, srv863413042=93, srv2017443714=53, srv1369309796=19, srv942973226=96, srv1459346975=22, srv519594352=80, srv894713365=94, srv608979280=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:55:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:55:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:55:34,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:55:34,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:55:34,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:55:34,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:55:34,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:55:34,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:55:34,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:55:34,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:55:34,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:55:34,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:55:34,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:55:34,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:55:34,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:55:34,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:55:34,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:55:34,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:55:34,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:55:34,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:55:34,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:55:34,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:55:34,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:55:34,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:55:34,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-12 19:55:34,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-12 19:55:34,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-12 19:55:34,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-12 19:55:34,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-12 19:55:34,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-12 19:55:34,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-12 19:55:34,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-12 19:55:34,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-12 19:55:34,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-12 19:55:34,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-12 19:55:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-12 19:55:34,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-12 19:55:34,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-12 19:55:34,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-12 19:55:34,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-12 19:55:34,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-12 19:55:34,690 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:34,691 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-12 19:55:34,691 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-12 19:55:35,039 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 650 ms to try 5000 different iterations. Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17869121150098094. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949393048248971, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050734227505979695, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949393048248971, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:35,052 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1779048366:81 , srv959950567:81 , srv1019100841:82 , srv1273347606:82 , srv140636426:82 , srv2085880201:82 , srv519594352:82 , srv1234304578:83 , srv1295264407:83 , srv174313478:83 , srv1954349585:83 , srv633839878:83 , srv644108814:83 , srv1028640747:84 , srv1157518354:84 , srv1676044950:84 , srv1879025815:84 , srv294130555:84 , srv109068049:85 , srv1093378032:85 , srv1364600471:85 , srv1459346975:85 , srv1464173198:85 , srv1867076181:85 , srv1897318501:85 , srv197518247:85 , srv2141781224:85 , srv255787727:85 , srv270838116:85 , srv29235473:85 , srv326553882:85 , srv386706952:85 , srv628808200:85 , srv688958658:85 , srv776754724:85 , srv85359122:85 , srv1077778546:86 , srv1100511191:86 , srv1519996330:86 , srv1568715046:86 , srv1752299525:86 , srv1898567124:86 , srv1976755873:86 , srv338480292:86 , srv378746305:86 , srv438166455:86 , srv495629854:86 , srv675540463:86 , srv766081533:86 , srv863413042:86 , srv917088438:86 , srv1095906840:87 , srv1149519425:87 , srv128293127:87 , srv1410391633:87 , srv1497966014:87 , srv1698727444:87 , srv1797824579:87 , srv1820663568:87 , srv188151044:87 , srv2014372614:87 , srv2017443714:87 , srv331537872:87 , srv356795897:87 , srv360329639:87 , srv39823001:87 , srv409035089:87 , srv480137467:87 , srv482336294:87 , srv958416556:87 , srv1303964407:88 , srv1478180453:88 , srv1483293538:88 , srv2030114690:88 , srv608979280:88 , srv626940684:88 , srv894713365:88 , srv942973226:88 , srv1190916849:89 , srv1690564732:89 , srv1890067481:89 , srv1983324433:89 , srv218522795:89 , srv235306761:89 , srv257264350:89 , srv518376412:89 , srv991879183:89 , srv1070449500:90 , srv1528144278:90 , srv1847830882:90 , srv2044969995:90 , srv580537614:90 , srv1501516420:91 , srv1935580265:91 , srv81178558:91 , srv1072264840:92 , srv1133207750:92 , srv1920782173:92 , srv2085646529:92 , srv1369309796:441 } 2023-12-12 19:55:35,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:55:35,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv675540463=87, srv1920782173=46, srv2085880201=57, srv1519996330=28, srv1879025815=41, srv29235473=64, srv438166455=75, srv1100511191=8, srv1847830882=39, srv1028640747=1, srv386706952=72, srv1364600471=18, srv1478180453=24, srv174313478=34, srv1779048366=36, srv85359122=92, srv633839878=85, srv958416556=97, srv2141781224=58, srv2085646529=56, srv1133207750=9, srv1528144278=29, srv1483293538=25, srv1752299525=35, srv1273347606=14, srv235306761=60, srv626940684=83, srv1157518354=11, srv1072264840=3, srv1410391633=21, srv294130555=65, srv766081533=89, srv1303964407=17, srv1797824579=37, srv197518247=49, srv326553882=66, srv482336294=77, srv1820663568=38, srv1077778546=4, srv917088438=95, srv1019100841=0, srv1890067481=43, srv270838116=63, srv991879183=99, srv255787727=61, srv1676044950=31, srv331537872=67, srv1983324433=51, srv409035089=74, srv1935580265=47, srv39823001=73, srv518376412=79, srv356795897=69, srv580537614=81, srv688958658=88, srv959950567=98, srv257264350=62, srv1070449500=2, srv1501516420=27, srv1898567124=45, srv1149519425=10, srv338480292=68, srv776754724=90, srv1698727444=33, srv2044969995=55, srv644108814=86, srv2014372614=52, srv1867076181=40, srv378746305=71, srv1497966014=26, srv2030114690=54, srv1954349585=48, srv128293127=15, srv81178558=91, srv1295264407=16, srv495629854=78, srv628808200=84, srv1093378032=6, srv1690564732=32, srv1568715046=30, srv1234304578=13, srv188151044=42, srv480137467=76, srv1897318501=44, srv1095906840=7, srv360329639=70, srv140636426=20, srv218522795=59, srv1190916849=12, srv1976755873=50, srv109068049=5, srv1464173198=23, srv863413042=93, srv2017443714=53, srv1369309796=19, srv942973226=96, srv1459346975=22, srv519594352=80, srv894713365=94, srv608979280=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:55:35,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:55:35,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:55:35,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:55:35,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:55:35,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:55:35,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:55:35,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:55:35,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:55:35,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:55:35,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:55:35,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-12 19:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-12 19:55:35,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-12 19:55:35,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-12 19:55:35,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-12 19:55:35,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-12 19:55:35,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-12 19:55:35,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-12 19:55:35,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-12 19:55:35,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-12 19:55:35,235 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949393048248971, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050734227505979695, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949393048248971, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:35,236 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-12 19:55:35,236 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1786583154018785, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949393048248971, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050734227505979695, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949393048248971, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-12 19:55:35,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 319 ms to try 5000 different iterations. Found a solution that moves 2607 regions; Going from a computed imbalance of 0.1786583154018785 to a new imbalance of 0.013985356133386037. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021713595612960955); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032947259150373734, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2896666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021713595612960955); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:35,441 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2085646529:79 , srv1072264840:80 , srv1133207750:80 , srv1501516420:80 , srv1920782173:80 , srv1847830882:81 , srv257264350:81 , srv81178558:81 , srv1483293538:82 , srv1528144278:82 , srv1890067481:82 , srv2044969995:82 , srv580537614:82 , srv942973226:82 , srv1190916849:83 , srv128293127:83 , srv1478180453:83 , srv1698727444:83 , srv1935580265:83 , srv1983324433:83 , srv218522795:83 , srv331537872:83 , srv518376412:83 , srv626940684:83 , srv991879183:83 , srv1070449500:84 , srv1100511191:84 , srv1149519425:84 , srv1690564732:84 , srv1898567124:84 , srv2030114690:84 , srv235306761:84 , srv356795897:84 , srv360329639:84 , srv438166455:84 , srv766081533:84 , srv1303964407:85 , srv1820663568:85 , srv188151044:85 , srv1976755873:85 , srv2014372614:85 , srv338480292:85 , srv39823001:85 , srv480137467:85 , srv482336294:85 , srv675540463:85 , srv958416556:85 , srv1077778546:86 , srv1093378032:86 , srv1497966014:86 , srv1519996330:86 , srv1568715046:86 , srv1752299525:86 , srv2017443714:86 , srv608979280:86 , srv776754724:86 , srv917088438:86 , srv1028640747:87 , srv109068049:87 , srv1364600471:87 , srv1410391633:87 , srv1464173198:87 , srv1797824579:87 , srv1867076181:87 , srv1954349585:87 , srv2141781224:87 , srv326553882:87 , srv378746305:87 , srv409035089:87 , srv628808200:87 , srv85359122:87 , srv863413042:87 , srv1095906840:88 , srv1157518354:88 , srv1234304578:88 , srv1676044950:88 , srv174313478:88 , srv1879025815:88 , srv1897318501:88 , srv255787727:88 , srv270838116:88 , srv294130555:88 , srv495629854:88 , srv519594352:88 , srv688958658:88 , srv1295264407:89 , srv1459346975:89 , srv1779048366:89 , srv197518247:89 , srv386706952:89 , srv633839878:89 , srv29235473:90 , srv644108814:90 , srv894713365:90 , srv959950567:90 , srv1019100841:91 , srv1273347606:91 , srv140636426:91 , srv2085880201:92 , srv1369309796:526 } 2023-12-12 19:55:35,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:55:35,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv675540463=87, srv1920782173=46, srv2085880201=57, srv1519996330=28, srv1879025815=41, srv29235473=64, srv438166455=75, srv1100511191=8, srv1847830882=39, srv1028640747=1, srv386706952=72, srv1364600471=18, srv1478180453=24, srv174313478=34, srv1779048366=36, srv85359122=92, srv633839878=85, srv958416556=97, srv2141781224=58, srv2085646529=56, srv1133207750=9, srv1528144278=29, srv1483293538=25, srv1752299525=35, srv1273347606=14, srv235306761=60, srv626940684=83, srv1157518354=11, srv1072264840=3, srv1410391633=21, srv294130555=65, srv766081533=89, srv1303964407=17, srv1797824579=37, srv197518247=49, srv326553882=66, srv482336294=77, srv1820663568=38, srv1077778546=4, srv917088438=95, srv1019100841=0, srv1890067481=43, srv270838116=63, srv991879183=99, srv255787727=61, srv1676044950=31, srv331537872=67, srv1983324433=51, srv409035089=74, srv1935580265=47, srv39823001=73, srv518376412=79, srv356795897=69, srv580537614=81, srv688958658=88, srv959950567=98, srv257264350=62, srv1070449500=2, srv1501516420=27, srv1898567124=45, srv1149519425=10, srv338480292=68, srv776754724=90, srv1698727444=33, srv2044969995=55, srv644108814=86, srv2014372614=52, srv1867076181=40, srv378746305=71, srv1497966014=26, srv2030114690=54, srv1954349585=48, srv128293127=15, srv81178558=91, srv1295264407=16, srv495629854=78, srv628808200=84, srv1093378032=6, srv1690564732=32, srv1568715046=30, srv1234304578=13, srv188151044=42, srv480137467=76, srv1897318501=44, srv1095906840=7, srv360329639=70, srv140636426=20, srv218522795=59, srv1190916849=12, srv1976755873=50, srv109068049=5, srv1464173198=23, srv863413042=93, srv2017443714=53, srv1369309796=19, srv942973226=96, srv1459346975=22, srv519594352=80, srv894713365=94, srv608979280=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:55:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:55:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:55:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-12 19:55:35,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-12 19:55:35,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-12 19:55:35,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-12 19:55:35,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-12 19:55:35,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-12 19:55:35,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-12 19:55:35,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-12 19:55:35,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-12 19:55:35,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-12 19:55:35,596 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021713595612960955); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032947259150373734, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021713595612960955); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:35,596 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-12 19:55:35,596 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013967101529222559, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021713595612960955); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032947259150373734, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021713595612960955); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-12 19:55:35,801 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 301 ms to try 5000 different iterations. Found a solution that moves 1073 regions; Going from a computed imbalance of 0.013967101529222559 to a new imbalance of 7.040865767608551E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054843927113124155); MoveCostFunction : (multiplier=7.0, imbalance=0.11922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:35,804 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv894713365:79 , srv1095906840:82 , srv1070449500:83 , srv1410391633:83 , srv1935580265:83 , srv197518247:83 , srv2085880201:83 , srv608979280:83 , srv1273347606:84 , srv1459346975:84 , srv1497966014:84 , srv1528144278:84 , srv1797824579:84 , srv2044969995:84 , srv255787727:84 , srv270838116:84 , srv29235473:84 , srv378746305:84 , srv386706952:84 , srv409035089:84 , srv495629854:84 , srv644108814:84 , srv863413042:84 , srv1019100841:85 , srv1072264840:85 , srv109068049:85 , srv1364600471:85 , srv140636426:85 , srv1464173198:85 , srv1676044950:85 , srv1690564732:85 , srv1820663568:85 , srv1879025815:85 , srv188151044:85 , srv1897318501:85 , srv1920782173:85 , srv1983324433:85 , srv2014372614:85 , srv2017443714:85 , srv2141781224:85 , srv218522795:85 , srv518376412:85 , srv633839878:85 , srv688958658:85 , srv81178558:85 , srv85359122:85 , srv917088438:85 , srv991879183:85 , srv1077778546:86 , srv1093378032:86 , srv1133207750:86 , srv1157518354:86 , srv1190916849:86 , srv1234304578:86 , srv1295264407:86 , srv1303964407:86 , srv1519996330:86 , srv1568715046:86 , srv1752299525:86 , srv1890067481:86 , srv2030114690:86 , srv235306761:86 , srv294130555:86 , srv326553882:86 , srv360329639:86 , srv39823001:86 , srv480137467:86 , srv482336294:86 , srv580537614:86 , srv626940684:86 , srv628808200:86 , srv958416556:86 , srv959950567:86 , srv1028640747:87 , srv1100511191:87 , srv1149519425:87 , srv128293127:87 , srv1478180453:87 , srv1501516420:87 , srv1698727444:87 , srv174313478:87 , srv1847830882:87 , srv1867076181:87 , srv1898567124:87 , srv1954349585:87 , srv1976755873:87 , srv2085646529:87 , srv257264350:87 , srv331537872:87 , srv338480292:87 , srv356795897:87 , srv438166455:87 , srv675540463:87 , srv776754724:87 , srv942973226:87 , srv1483293538:88 , srv1779048366:88 , srv519594352:88 , srv766081533:88 , srv1369309796:540 } 2023-12-12 19:55:35,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:55:35,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv675540463=87, srv1920782173=46, srv2085880201=57, srv1519996330=28, srv1879025815=41, srv29235473=64, srv438166455=75, srv1100511191=8, srv1847830882=39, srv1028640747=1, srv386706952=72, srv1364600471=18, srv1478180453=24, srv174313478=34, srv1779048366=36, srv85359122=92, srv633839878=85, srv958416556=97, srv2141781224=58, srv2085646529=56, srv1133207750=9, srv1528144278=29, srv1483293538=25, srv1752299525=35, srv1273347606=14, srv235306761=60, srv626940684=83, srv1157518354=11, srv1072264840=3, srv1410391633=21, srv294130555=65, srv766081533=89, srv1303964407=17, srv1797824579=37, srv197518247=49, srv326553882=66, srv482336294=77, srv1820663568=38, srv1077778546=4, srv917088438=95, srv1019100841=0, srv1890067481=43, srv270838116=63, srv991879183=99, srv255787727=61, srv1676044950=31, srv331537872=67, srv1983324433=51, srv409035089=74, srv1935580265=47, srv39823001=73, srv518376412=79, srv356795897=69, srv580537614=81, srv688958658=88, srv959950567=98, srv257264350=62, srv1070449500=2, srv1501516420=27, srv1898567124=45, srv1149519425=10, srv338480292=68, srv776754724=90, srv1698727444=33, srv2044969995=55, srv644108814=86, srv2014372614=52, srv1867076181=40, srv378746305=71, srv1497966014=26, srv2030114690=54, srv1954349585=48, srv128293127=15, srv81178558=91, srv1295264407=16, srv495629854=78, srv628808200=84, srv1093378032=6, srv1690564732=32, srv1568715046=30, srv1234304578=13, srv188151044=42, srv480137467=76, srv1897318501=44, srv1095906840=7, srv360329639=70, srv140636426=20, srv218522795=59, srv1190916849=12, srv1976755873=50, srv109068049=5, srv1464173198=23, srv863413042=93, srv2017443714=53, srv1369309796=19, srv942973226=96, srv1459346975=22, srv519594352=80, srv894713365=94, srv608979280=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:55:35,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:55:35,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:55:35,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:55:35,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:55:35,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:55:35,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-12 19:55:35,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-12 19:55:35,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-12 19:55:35,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-12 19:55:35,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-12 19:55:35,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-12 19:55:35,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-12 19:55:35,918 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054843927113124155); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:35,918 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-12 19:55:35,918 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.965732701757335E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054843927113124155); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-12 19:55:36,153 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 313 ms to try 5000 different iterations. Found a solution that moves 272 regions; Going from a computed imbalance of 6.965732701757335E-4 to a new imbalance of 1.0661142929028332E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.030222222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:36,154 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1303964407:83 , srv1519996330:83 , srv1867076181:83 , srv766081533:83 , srv1019100841:84 , srv1077778546:84 , srv1133207750:84 , srv1149519425:84 , srv1157518354:84 , srv1190916849:84 , srv1234304578:84 , srv1273347606:84 , srv140636426:84 , srv1459346975:84 , srv1478180453:84 , srv1483293538:84 , srv1501516420:84 , srv1568715046:84 , srv1690564732:84 , srv174313478:84 , srv1752299525:84 , srv1779048366:84 , srv1797824579:84 , srv1847830882:84 , srv1897318501:84 , srv1976755873:84 , srv2017443714:84 , srv2030114690:84 , srv2085646529:84 , srv235306761:84 , srv29235473:84 , srv294130555:84 , srv356795897:84 , srv378746305:84 , srv386706952:84 , srv39823001:84 , srv409035089:84 , srv480137467:84 , srv482336294:84 , srv495629854:84 , srv519594352:84 , srv580537614:84 , srv626940684:84 , srv628808200:84 , srv675540463:84 , srv688958658:84 , srv776754724:84 , srv958416556:84 , srv1028640747:85 , srv1070449500:85 , srv1072264840:85 , srv109068049:85 , srv1093378032:85 , srv1095906840:85 , srv1100511191:85 , srv128293127:85 , srv1295264407:85 , srv1364600471:85 , srv1410391633:85 , srv1464173198:85 , srv1497966014:85 , srv1676044950:85 , srv1698727444:85 , srv1820663568:85 , srv1879025815:85 , srv188151044:85 , srv1890067481:85 , srv1898567124:85 , srv1920782173:85 , srv1935580265:85 , srv1954349585:85 , srv197518247:85 , srv1983324433:85 , srv2014372614:85 , srv2044969995:85 , srv2085880201:85 , srv2141781224:85 , srv218522795:85 , srv255787727:85 , srv257264350:85 , srv270838116:85 , srv326553882:85 , srv331537872:85 , srv338480292:85 , srv360329639:85 , srv438166455:85 , srv518376412:85 , srv608979280:85 , srv633839878:85 , srv644108814:85 , srv81178558:85 , srv85359122:85 , srv863413042:85 , srv894713365:85 , srv917088438:85 , srv942973226:85 , srv959950567:85 , srv991879183:85 , srv1528144278:86 , srv1369309796:636 } 2023-12-12 19:55:36,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:55:36,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv675540463=87, srv1920782173=46, srv2085880201=57, srv1519996330=28, srv1879025815=41, srv29235473=64, srv438166455=75, srv1100511191=8, srv1847830882=39, srv1028640747=1, srv386706952=72, srv1364600471=18, srv1478180453=24, srv174313478=34, srv1779048366=36, srv85359122=92, srv633839878=85, srv958416556=97, srv2141781224=58, srv2085646529=56, srv1133207750=9, srv1528144278=29, srv1483293538=25, srv1752299525=35, srv1273347606=14, srv235306761=60, srv626940684=83, srv1157518354=11, srv1072264840=3, srv1410391633=21, srv294130555=65, srv766081533=89, srv1303964407=17, srv1797824579=37, srv197518247=49, srv326553882=66, srv482336294=77, srv1820663568=38, srv1077778546=4, srv917088438=95, srv1019100841=0, srv1890067481=43, srv270838116=63, srv991879183=99, srv255787727=61, srv1676044950=31, srv331537872=67, srv1983324433=51, srv409035089=74, srv1935580265=47, srv39823001=73, srv518376412=79, srv356795897=69, srv580537614=81, srv688958658=88, srv959950567=98, srv257264350=62, srv1070449500=2, srv1501516420=27, srv1898567124=45, srv1149519425=10, srv338480292=68, srv776754724=90, srv1698727444=33, srv2044969995=55, srv644108814=86, srv2014372614=52, srv1867076181=40, srv378746305=71, srv1497966014=26, srv2030114690=54, srv1954349585=48, srv128293127=15, srv81178558=91, srv1295264407=16, srv495629854=78, srv628808200=84, srv1093378032=6, srv1690564732=32, srv1568715046=30, srv1234304578=13, srv188151044=42, srv480137467=76, srv1897318501=44, srv1095906840=7, srv360329639=70, srv140636426=20, srv218522795=59, srv1190916849=12, srv1976755873=50, srv109068049=5, srv1464173198=23, srv863413042=93, srv2017443714=53, srv1369309796=19, srv942973226=96, srv1459346975=22, srv519594352=80, srv894713365=94, srv608979280=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:55:36,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:55:36,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-12 19:55:36,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-12 19:55:36,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-12 19:55:36,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-12 19:55:36,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-12 19:55:36,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-12 19:55:36,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-12 19:55:36,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-12 19:55:36,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-12 19:55:36,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.75655822152313E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:36,259 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1303964407:83 , srv1519996330:83 , srv1867076181:83 , srv766081533:83 , srv1019100841:84 , srv1077778546:84 , srv1133207750:84 , srv1149519425:84 , srv1157518354:84 , srv1190916849:84 , srv1234304578:84 , srv1273347606:84 , srv140636426:84 , srv1459346975:84 , srv1478180453:84 , srv1483293538:84 , srv1501516420:84 , srv1568715046:84 , srv1690564732:84 , srv174313478:84 , srv1752299525:84 , srv1779048366:84 , srv1797824579:84 , srv1847830882:84 , srv1897318501:84 , srv1976755873:84 , srv2017443714:84 , srv2030114690:84 , srv2085646529:84 , srv235306761:84 , srv29235473:84 , srv294130555:84 , srv356795897:84 , srv378746305:84 , srv386706952:84 , srv39823001:84 , srv409035089:84 , srv480137467:84 , srv482336294:84 , srv495629854:84 , srv519594352:84 , srv580537614:84 , srv626940684:84 , srv628808200:84 , srv675540463:84 , srv688958658:84 , srv776754724:84 , srv958416556:84 , srv1028640747:85 , srv1070449500:85 , srv1072264840:85 , srv109068049:85 , srv1093378032:85 , srv1095906840:85 , srv1100511191:85 , srv128293127:85 , srv1295264407:85 , srv1364600471:85 , srv1410391633:85 , srv1464173198:85 , srv1497966014:85 , srv1676044950:85 , srv1698727444:85 , srv1820663568:85 , srv1879025815:85 , srv188151044:85 , srv1890067481:85 , srv1898567124:85 , srv1920782173:85 , srv1935580265:85 , srv1954349585:85 , srv197518247:85 , srv1983324433:85 , srv2014372614:85 , srv2044969995:85 , srv2085880201:85 , srv2141781224:85 , srv218522795:85 , srv255787727:85 , srv257264350:85 , srv270838116:85 , srv326553882:85 , srv331537872:85 , srv338480292:85 , srv360329639:85 , srv438166455:85 , srv518376412:85 , srv608979280:85 , srv633839878:85 , srv644108814:85 , srv81178558:85 , srv85359122:85 , srv863413042:85 , srv894713365:85 , srv917088438:85 , srv942973226:85 , srv959950567:85 , srv991879183:85 , srv1528144278:86 , srv1369309796:636 } 2023-12-12 19:55:36,283 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=260 (was 260), ProcessCount=174 (was 174), AvailableMemoryMB=3528 (was 4031) 2023-12-12 19:55:36,290 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=174, AvailableMemoryMB=3528 2023-12-12 19:55:36,291 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-12 19:55:36,291 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-12 19:55:36,301 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1058396517:84 , srv1245539530:84 , srv1272112418:84 , srv1378913562:84 , srv1601666591:84 , srv1633618649:84 , srv1661484977:84 , srv1672733158:84 , srv1696519127:84 , srv1761913587:84 , srv1790335697:84 , srv1802984138:84 , srv1912355687:84 , srv1913481216:84 , srv1976028794:84 , srv2086501053:84 , srv2111196295:84 , srv230852378:84 , srv464543713:84 , srv486376184:84 , srv493747083:84 , srv555389261:84 , srv603971373:84 , srv645300159:84 , srv667648205:84 , srv744038970:84 , srv892005062:84 , srv944556259:84 , srv975818921:84 , srv1266878009:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-12 19:55:36,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:55:36,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1058396517=0, srv2086501053=16, srv1913481216=14, srv1912355687=13, srv2111196295=17, srv464543713=19, srv1761913587=10, srv944556259=28, srv1601666591=5, srv1266878009=2, srv603971373=23, srv1661484977=7, srv975818921=29, srv667648205=25, srv555389261=22, srv493747083=21, srv230852378=18, srv1696519127=9, srv892005062=27, srv486376184=20, srv1802984138=12, srv1672733158=8, srv1378913562=4, srv1272112418=3, srv1790335697=11, srv645300159=24, srv1633618649=6, srv744038970=26, srv1976028794=15, srv1245539530=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:55:36,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:55:36,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:55:36,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:55:36,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:55:36,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:55:36,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:55:36,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:55:36,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:55:36,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:55:36,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:55:36,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-12 19:55:36,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0662081096862555, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:55:36,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912401662856009, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0662081096862555, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-12 19:57:36,323 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9483909 different iterations. Found a solution that moves 1832 regions; Going from a computed imbalance of 0.9912401662856009 to a new imbalance of 4.275979392340115E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6785185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:57:36,328 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1058396517:90 , srv1245539530:90 , srv1266878009:90 , srv1272112418:90 , srv1378913562:90 , srv1601666591:90 , srv1633618649:90 , srv1661484977:90 , srv1672733158:90 , srv1696519127:90 , srv1761913587:90 , srv1790335697:90 , srv1802984138:90 , srv1912355687:90 , srv1913481216:90 , srv1976028794:90 , srv2086501053:90 , srv2111196295:90 , srv230852378:90 , srv464543713:90 , srv486376184:90 , srv493747083:90 , srv555389261:90 , srv603971373:90 , srv645300159:90 , srv667648205:90 , srv744038970:90 , srv892005062:90 , srv944556259:90 , srv975818921:90 } 2023-12-12 19:57:36,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-12 19:57:36,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1058396517=0, srv2086501053=16, srv1913481216=14, srv1912355687=13, srv2111196295=17, srv464543713=19, srv1761913587=10, srv944556259=28, srv1601666591=5, srv1266878009=2, srv603971373=23, srv1661484977=7, srv975818921=29, srv667648205=25, srv555389261=22, srv493747083=21, srv230852378=18, srv1696519127=9, srv892005062=27, srv486376184=20, srv1802984138=12, srv1672733158=8, srv1378913562=4, srv1272112418=3, srv1790335697=11, srv645300159=24, srv1633618649=6, srv744038970=26, srv1976028794=15, srv1245539530=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-12 19:57:36,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-12 19:57:36,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-12 19:57:36,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-12 19:57:36,342 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1058396517:90 , srv1245539530:90 , srv1266878009:90 , srv1272112418:90 , srv1378913562:90 , srv1601666591:90 , srv1633618649:90 , srv1661484977:90 , srv1672733158:90 , srv1696519127:90 , srv1761913587:90 , srv1790335697:90 , srv1802984138:90 , srv1912355687:90 , srv1913481216:90 , srv1976028794:90 , srv2086501053:90 , srv2111196295:90 , srv230852378:90 , srv464543713:90 , srv486376184:90 , srv493747083:90 , srv555389261:90 , srv603971373:90 , srv645300159:90 , srv667648205:90 , srv744038970:90 , srv892005062:90 , srv944556259:90 , srv975818921:90 } 2023-12-12 19:57:36,353 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=181 (was 260), ProcessCount=168 (was 174), AvailableMemoryMB=4375 (was 3528) - AvailableMemoryMB LEAK? -