2023-03-06 12:17:25,550 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-06 12:17:25,866 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-06 12:17:25,984 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-06 12:17:26,055 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-06 12:17:26,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-06 12:17:26,082 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=122, ProcessCount=160, AvailableMemoryMB=10169 2023-03-06 12:17:26,084 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-06 12:17:26,085 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-06 12:17:26,229 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1036071240:84 , srv104425584:84 , srv1061548863:84 , srv1070240337:84 , srv1080827724:84 , srv1159527066:84 , srv1195190387:84 , srv1198585277:84 , srv1203842989:84 , srv1229761884:84 , srv1272796615:84 , srv1298587597:84 , srv13023610:84 , srv1319165979:84 , srv1345504288:84 , srv1348267926:84 , srv1351993416:84 , srv1358650438:84 , srv1363215452:84 , srv1404786688:84 , srv1418563084:84 , srv1428772697:84 , srv1441230777:84 , srv1478909148:84 , srv1486139982:84 , srv1487458406:84 , srv1519202675:84 , srv1594453052:84 , srv1622891164:84 , srv1623524262:84 , srv162598068:84 , srv1631900979:84 , srv16393541:84 , srv1652974816:84 , srv1667650639:84 , srv1700741928:84 , srv1707768104:84 , srv1720944665:84 , srv1750898589:84 , srv1818559601:84 , srv1828301754:84 , srv1901434085:84 , srv1921155863:84 , srv1977546294:84 , srv1981746023:84 , srv2011988751:84 , srv2014826720:84 , srv2017674752:84 , srv2049315006:84 , srv2059309922:84 , srv206065548:84 , srv2091361425:84 , srv2092732436:84 , srv2100802429:84 , srv2106949010:84 , srv2111047790:84 , srv211318396:84 , srv2114117980:84 , srv2117728392:84 , srv2119060764:84 , srv2135325644:84 , srv2135386682:84 , srv2137873411:84 , srv226488473:84 , srv236764756:84 , srv260781321:84 , srv264479876:84 , srv270394993:84 , srv324458814:84 , srv325805454:84 , srv33744095:84 , srv341716996:84 , srv34274117:84 , srv389287954:84 , srv407240095:84 , srv41011186:84 , srv423962682:84 , srv425940983:84 , srv441235303:84 , srv505095366:84 , srv506930456:84 , srv554342822:84 , srv559625764:84 , srv626906301:84 , srv656185783:84 , srv667244869:84 , srv755142258:84 , srv783452215:84 , srv787217009:84 , srv824017092:84 , srv887898934:84 , srv890554055:84 , srv915544959:84 , srv916491025:84 , srv930776292:84 , srv937739140:84 , srv956818730:84 , srv958395246:84 , srv98143947:84 , srv1771971753:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-06 12:17:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:26,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv626906301=84, srv104425584=1, srv2049315006=49, srv407240095=75, srv260781321=66, srv1351993416=16, srv1363215452=18, srv2011988751=46, srv2135325644=61, srv1404786688=19, srv937739140=96, srv211318396=57, srv656185783=85, srv41011186=76, srv958395246=98, srv1418563084=20, srv236764756=65, srv1298587597=11, srv425940983=78, srv1198585277=7, srv341716996=72, srv787217009=89, srv755142258=87, srv2111047790=56, srv1750898589=38, srv1070240337=3, srv2137873411=63, srv783452215=88, srv1358650438=17, srv270394993=68, srv1348267926=15, srv956818730=97, srv1622891164=28, srv2059309922=50, srv554342822=82, srv264479876=67, srv206065548=51, srv916491025=94, srv1667650639=34, srv1080827724=4, srv1921155863=43, srv2135386682=62, srv505095366=80, srv1700741928=35, srv887898934=91, srv1707768104=36, srv2119060764=60, srv1901434085=42, srv389287954=74, srv1272796615=10, srv667244869=86, srv325805454=70, srv162598068=30, srv890554055=92, srv1061548863=2, srv1345504288=14, srv1818559601=40, srv1981746023=45, srv1441230777=22, srv324458814=69, srv1486139982=24, srv2092732436=53, srv915544959=93, srv1594453052=27, srv1631900979=31, srv1319165979=13, srv1478909148=23, srv1623524262=29, srv2014826720=47, srv1828301754=41, srv1159527066=5, srv16393541=32, srv2114117980=58, srv33744095=71, srv930776292=95, srv1428772697=21, srv1487458406=25, srv2091361425=52, srv34274117=73, srv441235303=79, srv2117728392=59, srv559625764=83, srv1229761884=9, srv2017674752=48, srv1652974816=33, srv2106949010=55, srv1977546294=44, srv1771971753=39, srv1036071240=0, srv1519202675=26, srv226488473=64, srv13023610=12, srv423962682=77, srv98143947=99, srv1195190387=6, srv1720944665=37, srv2100802429=54, srv1203842989=8, srv824017092=90, srv506930456=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-06 12:17:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-06 12:17:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-06 12:17:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-06 12:17:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-06 12:17:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-06 12:17:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-06 12:17:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-06 12:17:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-06 12:17:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-06 12:17:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-06 12:17:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-06 12:17:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-06 12:17:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-06 12:17:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-06 12:17:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-06 12:17:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-06 12:17:26,526 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:26,526 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-06 12:17:26,527 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-06 12:17:26,806 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 428 ms to try 5000 different iterations. Found a solution that moves 4740 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1785085328751394. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891309171501667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05737427087531157, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5266666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891309171501667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24166666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:26,829 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1351993416:82 , srv1720944665:82 , srv236764756:82 , srv626906301:82 , srv1036071240:83 , srv1478909148:83 , srv2119060764:83 , srv264479876:83 , srv41011186:83 , srv1061548863:84 , srv1195190387:84 , srv1198585277:84 , srv1203842989:84 , srv1622891164:84 , srv1652974816:84 , srv1707768104:84 , srv1828301754:84 , srv2049315006:84 , srv505095366:84 , srv937739140:84 , srv1070240337:85 , srv1159527066:85 , srv1229761884:85 , srv1319165979:85 , srv1428772697:85 , srv16393541:85 , srv1981746023:85 , srv260781321:85 , srv667244869:85 , srv824017092:85 , srv916491025:85 , srv1080827724:86 , srv13023610:86 , srv1345504288:86 , srv1358650438:86 , srv1631900979:86 , srv1700741928:86 , srv1901434085:86 , srv2017674752:86 , srv2091361425:86 , srv2100802429:86 , srv211318396:86 , srv2117728392:86 , srv2135325644:86 , srv270394993:86 , srv425940983:86 , srv441235303:86 , srv506930456:86 , srv783452215:86 , srv787217009:86 , srv890554055:86 , srv958395246:86 , srv1298587597:87 , srv1348267926:87 , srv1519202675:87 , srv1623524262:87 , srv1921155863:87 , srv1977546294:87 , srv2014826720:87 , srv2135386682:87 , srv2137873411:87 , srv226488473:87 , srv423962682:87 , srv656185783:87 , srv755142258:87 , srv915544959:87 , srv930776292:87 , srv104425584:88 , srv1418563084:88 , srv1441230777:88 , srv1594453052:88 , srv1750898589:88 , srv1818559601:88 , srv2011988751:88 , srv206065548:88 , srv2114117980:88 , srv33744095:88 , srv389287954:88 , srv407240095:88 , srv1404786688:89 , srv1486139982:89 , srv162598068:89 , srv1667650639:89 , srv324458814:89 , srv325805454:89 , srv34274117:89 , srv559625764:89 , srv956818730:89 , srv1272796615:90 , srv1487458406:90 , srv2059309922:90 , srv2092732436:90 , srv2111047790:90 , srv341716996:90 , srv554342822:90 , srv887898934:90 , srv98143947:90 , srv2106949010:91 , srv1363215452:93 , srv1771971753:436 } 2023-03-06 12:17:26,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:26,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv626906301=84, srv104425584=1, srv2049315006=49, srv407240095=75, srv260781321=66, srv1351993416=16, srv1363215452=18, srv2011988751=46, srv2135325644=61, srv1404786688=19, srv937739140=96, srv211318396=57, srv656185783=85, srv41011186=76, srv958395246=98, srv1418563084=20, srv236764756=65, srv1298587597=11, srv425940983=78, srv1198585277=7, srv341716996=72, srv787217009=89, srv755142258=87, srv2111047790=56, srv1750898589=38, srv1070240337=3, srv2137873411=63, srv783452215=88, srv1358650438=17, srv270394993=68, srv1348267926=15, srv956818730=97, srv1622891164=28, srv2059309922=50, srv554342822=82, srv264479876=67, srv206065548=51, srv916491025=94, srv1667650639=34, srv1080827724=4, srv1921155863=43, srv2135386682=62, srv505095366=80, srv1700741928=35, srv887898934=91, srv1707768104=36, srv2119060764=60, srv1901434085=42, srv389287954=74, srv1272796615=10, srv667244869=86, srv325805454=70, srv162598068=30, srv890554055=92, srv1061548863=2, srv1345504288=14, srv1818559601=40, srv1981746023=45, srv1441230777=22, srv324458814=69, srv1486139982=24, srv2092732436=53, srv915544959=93, srv1594453052=27, srv1631900979=31, srv1319165979=13, srv1478909148=23, srv1623524262=29, srv2014826720=47, srv1828301754=41, srv1159527066=5, srv16393541=32, srv2114117980=58, srv33744095=71, srv930776292=95, srv1428772697=21, srv1487458406=25, srv2091361425=52, srv34274117=73, srv441235303=79, srv2117728392=59, srv559625764=83, srv1229761884=9, srv2017674752=48, srv1652974816=33, srv2106949010=55, srv1977546294=44, srv1771971753=39, srv1036071240=0, srv1519202675=26, srv226488473=64, srv13023610=12, srv423962682=77, srv98143947=99, srv1195190387=6, srv1720944665=37, srv2100802429=54, srv1203842989=8, srv824017092=90, srv506930456=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:26,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:26,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:26,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:26,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:26,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-06 12:17:26,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-06 12:17:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-06 12:17:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-06 12:17:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-06 12:17:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-06 12:17:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-06 12:17:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-06 12:17:26,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891309171501667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05737427087531157, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891309171501667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24166666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:26,996 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-06 12:17:26,997 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17847534268575127, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891309171501667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05737427087531157, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891309171501667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24166666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-06 12:17:27,293 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations. Found a solution that moves 2659 regions; Going from a computed imbalance of 0.17847534268575127 to a new imbalance of 0.012865908865284518. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021602468994692866); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032628920678540146, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29544444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021602468994692866); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:27,304 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1363215452:77 , srv2106949010:80 , srv956818730:80 , srv1272796615:81 , srv1487458406:81 , srv1667650639:81 , srv887898934:82 , srv98143947:82 , srv1298587597:83 , srv1348267926:83 , srv1404786688:83 , srv1441230777:83 , srv1486139982:83 , srv1594453052:83 , srv1631900979:83 , srv1750898589:83 , srv1818559601:83 , srv2011988751:83 , srv2092732436:83 , srv2111047790:83 , srv2137873411:83 , srv226488473:83 , srv341716996:83 , srv407240095:83 , srv554342822:83 , srv559625764:83 , srv1418563084:84 , srv1519202675:84 , srv2059309922:84 , srv2114117980:84 , srv324458814:84 , srv33744095:84 , srv389287954:84 , srv783452215:84 , srv104425584:85 , srv1229761884:85 , srv1358650438:85 , srv162598068:85 , srv1921155863:85 , srv2017674752:85 , srv206065548:85 , srv2100802429:85 , srv2117728392:85 , srv2135325644:85 , srv270394993:85 , srv441235303:85 , srv915544959:85 , srv1070240337:86 , srv1428772697:86 , srv16393541:86 , srv1828301754:86 , srv1901434085:86 , srv1977546294:86 , srv2014826720:86 , srv211318396:86 , srv325805454:86 , srv34274117:86 , srv423962682:86 , srv656185783:86 , srv755142258:86 , srv890554055:86 , srv930776292:86 , srv1061548863:87 , srv1080827724:87 , srv1198585277:87 , srv1319165979:87 , srv1345504288:87 , srv1623524262:87 , srv1707768104:87 , srv2091361425:87 , srv787217009:87 , srv937739140:87 , srv958395246:87 , srv1159527066:88 , srv1203842989:88 , srv13023610:88 , srv1478909148:88 , srv1622891164:88 , srv1700741928:88 , srv1981746023:88 , srv2049315006:88 , srv425940983:88 , srv506930456:88 , srv626906301:88 , srv667244869:88 , srv916491025:88 , srv1036071240:89 , srv1351993416:89 , srv1652974816:89 , srv2135386682:89 , srv260781321:89 , srv505095366:89 , srv824017092:89 , srv1195190387:90 , srv1720944665:90 , srv2119060764:90 , srv264479876:90 , srv41011186:90 , srv236764756:91 , srv1771971753:530 } 2023-03-06 12:17:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv626906301=84, srv104425584=1, srv2049315006=49, srv407240095=75, srv260781321=66, srv1351993416=16, srv1363215452=18, srv2011988751=46, srv2135325644=61, srv1404786688=19, srv937739140=96, srv211318396=57, srv656185783=85, srv41011186=76, srv958395246=98, srv1418563084=20, srv236764756=65, srv1298587597=11, srv425940983=78, srv1198585277=7, srv341716996=72, srv787217009=89, srv755142258=87, srv2111047790=56, srv1750898589=38, srv1070240337=3, srv2137873411=63, srv783452215=88, srv1358650438=17, srv270394993=68, srv1348267926=15, srv956818730=97, srv1622891164=28, srv2059309922=50, srv554342822=82, srv264479876=67, srv206065548=51, srv916491025=94, srv1667650639=34, srv1080827724=4, srv1921155863=43, srv2135386682=62, srv505095366=80, srv1700741928=35, srv887898934=91, srv1707768104=36, srv2119060764=60, srv1901434085=42, srv389287954=74, srv1272796615=10, srv667244869=86, srv325805454=70, srv162598068=30, srv890554055=92, srv1061548863=2, srv1345504288=14, srv1818559601=40, srv1981746023=45, srv1441230777=22, srv324458814=69, srv1486139982=24, srv2092732436=53, srv915544959=93, srv1594453052=27, srv1631900979=31, srv1319165979=13, srv1478909148=23, srv1623524262=29, srv2014826720=47, srv1828301754=41, srv1159527066=5, srv16393541=32, srv2114117980=58, srv33744095=71, srv930776292=95, srv1428772697=21, srv1487458406=25, srv2091361425=52, srv34274117=73, srv441235303=79, srv2117728392=59, srv559625764=83, srv1229761884=9, srv2017674752=48, srv1652974816=33, srv2106949010=55, srv1977546294=44, srv1771971753=39, srv1036071240=0, srv1519202675=26, srv226488473=64, srv13023610=12, srv423962682=77, srv98143947=99, srv1195190387=6, srv1720944665=37, srv2100802429=54, srv1203842989=8, srv824017092=90, srv506930456=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:27,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:27,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:27,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:27,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:27,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-06 12:17:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-06 12:17:27,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-06 12:17:27,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-06 12:17:27,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-06 12:17:27,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-06 12:17:27,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-06 12:17:27,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-06 12:17:27,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-06 12:17:27,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-06 12:17:27,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-06 12:17:27,480 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021602468994692866); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032628920678540146, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021602468994692866); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:27,480 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-06 12:17:27,480 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012847290149338724, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021602468994692866); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032628920678540146, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021602468994692866); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-06 12:17:27,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 421 ms to try 5000 different iterations. Found a solution that moves 973 regions; Going from a computed imbalance of 0.012847290149338724 to a new imbalance of 6.171114487579263E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006215564820726548); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.10811111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006215564820726548); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:27,784 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2135386682:81 , srv325805454:82 , srv162598068:83 , srv1700741928:83 , srv260781321:83 , srv34274117:83 , srv506930456:83 , srv104425584:84 , srv1195190387:84 , srv13023610:84 , srv1345504288:84 , srv1623524262:84 , srv1652974816:84 , srv1901434085:84 , srv1981746023:84 , srv2014826720:84 , srv2059309922:84 , srv2092732436:84 , srv2111047790:84 , srv211318396:84 , srv264479876:84 , srv341716996:84 , srv425940983:84 , srv559625764:84 , srv755142258:84 , srv787217009:84 , srv824017092:84 , srv1159527066:85 , srv1319165979:85 , srv1418563084:85 , srv1486139982:85 , srv1622891164:85 , srv1921155863:85 , srv2049315006:85 , srv206065548:85 , srv2117728392:85 , srv2119060764:85 , srv236764756:85 , srv324458814:85 , srv33744095:85 , srv41011186:85 , srv423962682:85 , srv505095366:85 , srv554342822:85 , srv656185783:85 , srv667244869:85 , srv890554055:85 , srv915544959:85 , srv916491025:85 , srv930776292:85 , srv958395246:85 , srv98143947:85 , srv1036071240:86 , srv1061548863:86 , srv1070240337:86 , srv1080827724:86 , srv1198585277:86 , srv1203842989:86 , srv1358650438:86 , srv1404786688:86 , srv1428772697:86 , srv1478909148:86 , srv1487458406:86 , srv1594453052:86 , srv1720944665:86 , srv1750898589:86 , srv1977546294:86 , srv2011988751:86 , srv2091361425:86 , srv2106949010:86 , srv2114117980:86 , srv2135325644:86 , srv270394993:86 , srv389287954:86 , srv407240095:86 , srv887898934:86 , srv937739140:86 , srv1229761884:87 , srv1272796615:87 , srv1298587597:87 , srv1348267926:87 , srv1363215452:87 , srv1441230777:87 , srv1519202675:87 , srv16393541:87 , srv1667650639:87 , srv1707768104:87 , srv1818559601:87 , srv1828301754:87 , srv2017674752:87 , srv2137873411:87 , srv441235303:87 , srv783452215:87 , srv1351993416:88 , srv1631900979:88 , srv2100802429:88 , srv226488473:88 , srv626906301:88 , srv956818730:88 , srv1771971753:547 } 2023-03-06 12:17:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv626906301=84, srv104425584=1, srv2049315006=49, srv407240095=75, srv260781321=66, srv1351993416=16, srv1363215452=18, srv2011988751=46, srv2135325644=61, srv1404786688=19, srv937739140=96, srv211318396=57, srv656185783=85, srv41011186=76, srv958395246=98, srv1418563084=20, srv236764756=65, srv1298587597=11, srv425940983=78, srv1198585277=7, srv341716996=72, srv787217009=89, srv755142258=87, srv2111047790=56, srv1750898589=38, srv1070240337=3, srv2137873411=63, srv783452215=88, srv1358650438=17, srv270394993=68, srv1348267926=15, srv956818730=97, srv1622891164=28, srv2059309922=50, srv554342822=82, srv264479876=67, srv206065548=51, srv916491025=94, srv1667650639=34, srv1080827724=4, srv1921155863=43, srv2135386682=62, srv505095366=80, srv1700741928=35, srv887898934=91, srv1707768104=36, srv2119060764=60, srv1901434085=42, srv389287954=74, srv1272796615=10, srv667244869=86, srv325805454=70, srv162598068=30, srv890554055=92, srv1061548863=2, srv1345504288=14, srv1818559601=40, srv1981746023=45, srv1441230777=22, srv324458814=69, srv1486139982=24, srv2092732436=53, srv915544959=93, srv1594453052=27, srv1631900979=31, srv1319165979=13, srv1478909148=23, srv1623524262=29, srv2014826720=47, srv1828301754=41, srv1159527066=5, srv16393541=32, srv2114117980=58, srv33744095=71, srv930776292=95, srv1428772697=21, srv1487458406=25, srv2091361425=52, srv34274117=73, srv441235303=79, srv2117728392=59, srv559625764=83, srv1229761884=9, srv2017674752=48, srv1652974816=33, srv2106949010=55, srv1977546294=44, srv1771971753=39, srv1036071240=0, srv1519202675=26, srv226488473=64, srv13023610=12, srv423962682=77, srv98143947=99, srv1195190387=6, srv1720944665=37, srv2100802429=54, srv1203842989=8, srv824017092=90, srv506930456=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-06 12:17:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-06 12:17:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-06 12:17:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-06 12:17:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-06 12:17:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-06 12:17:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-06 12:17:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-06 12:17:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-06 12:17:27,934 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006215564820726548); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006215564820726548); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:27,935 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-06 12:17:27,935 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.102983571387992E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006215564820726548); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006215564820726548); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-06 12:17:28,223 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 389 ms to try 5000 different iterations. Found a solution that moves 273 regions; Going from a computed imbalance of 6.102983571387992E-4 to a new imbalance of 4.459659830697746E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.030333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:28,224 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1080827724:83 , srv1351993416:83 , srv1977546294:83 , srv1036071240:84 , srv104425584:84 , srv1159527066:84 , srv1195190387:84 , srv1203842989:84 , srv1272796615:84 , srv13023610:84 , srv1404786688:84 , srv1441230777:84 , srv1478909148:84 , srv1519202675:84 , srv1623524262:84 , srv16393541:84 , srv1707768104:84 , srv1720944665:84 , srv1818559601:84 , srv2017674752:84 , srv2059309922:84 , srv206065548:84 , srv2091361425:84 , srv2100802429:84 , srv2106949010:84 , srv2114117980:84 , srv226488473:84 , srv236764756:84 , srv260781321:84 , srv324458814:84 , srv34274117:84 , srv389287954:84 , srv407240095:84 , srv41011186:84 , srv423962682:84 , srv425940983:84 , srv441235303:84 , srv505095366:84 , srv554342822:84 , srv626906301:84 , srv656185783:84 , srv667244869:84 , srv824017092:84 , srv887898934:84 , srv916491025:84 , srv930776292:84 , srv958395246:84 , srv98143947:84 , srv1061548863:85 , srv1070240337:85 , srv1198585277:85 , srv1229761884:85 , srv1298587597:85 , srv1319165979:85 , srv1345504288:85 , srv1348267926:85 , srv1358650438:85 , srv1363215452:85 , srv1418563084:85 , srv1428772697:85 , srv1486139982:85 , srv1487458406:85 , srv1594453052:85 , srv1622891164:85 , srv162598068:85 , srv1631900979:85 , srv1652974816:85 , srv1667650639:85 , srv1700741928:85 , srv1750898589:85 , srv1828301754:85 , srv1921155863:85 , srv1981746023:85 , srv2014826720:85 , srv2049315006:85 , srv2092732436:85 , srv2111047790:85 , srv2119060764:85 , srv2135325644:85 , srv2135386682:85 , srv2137873411:85 , srv264479876:85 , srv270394993:85 , srv325805454:85 , srv33744095:85 , srv341716996:85 , srv506930456:85 , srv755142258:85 , srv783452215:85 , srv787217009:85 , srv915544959:85 , srv937739140:85 , srv956818730:85 , srv1901434085:86 , srv2011988751:86 , srv211318396:86 , srv2117728392:86 , srv559625764:86 , srv890554055:87 , srv1771971753:629 } 2023-03-06 12:17:28,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:28,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv626906301=84, srv104425584=1, srv2049315006=49, srv407240095=75, srv260781321=66, srv1351993416=16, srv1363215452=18, srv2011988751=46, srv2135325644=61, srv1404786688=19, srv937739140=96, srv211318396=57, srv656185783=85, srv41011186=76, srv958395246=98, srv1418563084=20, srv236764756=65, srv1298587597=11, srv425940983=78, srv1198585277=7, srv341716996=72, srv787217009=89, srv755142258=87, srv2111047790=56, srv1750898589=38, srv1070240337=3, srv2137873411=63, srv783452215=88, srv1358650438=17, srv270394993=68, srv1348267926=15, srv956818730=97, srv1622891164=28, srv2059309922=50, srv554342822=82, srv264479876=67, srv206065548=51, srv916491025=94, srv1667650639=34, srv1080827724=4, srv1921155863=43, srv2135386682=62, srv505095366=80, srv1700741928=35, srv887898934=91, srv1707768104=36, srv2119060764=60, srv1901434085=42, srv389287954=74, srv1272796615=10, srv667244869=86, srv325805454=70, srv162598068=30, srv890554055=92, srv1061548863=2, srv1345504288=14, srv1818559601=40, srv1981746023=45, srv1441230777=22, srv324458814=69, srv1486139982=24, srv2092732436=53, srv915544959=93, srv1594453052=27, srv1631900979=31, srv1319165979=13, srv1478909148=23, srv1623524262=29, srv2014826720=47, srv1828301754=41, srv1159527066=5, srv16393541=32, srv2114117980=58, srv33744095=71, srv930776292=95, srv1428772697=21, srv1487458406=25, srv2091361425=52, srv34274117=73, srv441235303=79, srv2117728392=59, srv559625764=83, srv1229761884=9, srv2017674752=48, srv1652974816=33, srv2106949010=55, srv1977546294=44, srv1771971753=39, srv1036071240=0, srv1519202675=26, srv226488473=64, srv13023610=12, srv423962682=77, srv98143947=99, srv1195190387=6, srv1720944665=37, srv2100802429=54, srv1203842989=8, srv824017092=90, srv506930456=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:28,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:28,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-06 12:17:28,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-06 12:17:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-06 12:17:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-06 12:17:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-06 12:17:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-06 12:17:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-06 12:17:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-06 12:17:28,351 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:28,351 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-06 12:17:28,351 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.268501144981231E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-06 12:17:28,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 323 ms to try 5000 different iterations. Found a solution that moves 37 regions; Going from a computed imbalance of 4.268501144981231E-5 to a new imbalance of 7.948501337768768E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:28,578 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv890554055:82 , srv2011988751:83 , srv2091361425:83 , srv2100802429:83 , srv2119060764:83 , srv1036071240:84 , srv1061548863:84 , srv1070240337:84 , srv1080827724:84 , srv1159527066:84 , srv1195190387:84 , srv1198585277:84 , srv1203842989:84 , srv1229761884:84 , srv1272796615:84 , srv1298587597:84 , srv13023610:84 , srv1319165979:84 , srv1345504288:84 , srv1348267926:84 , srv1351993416:84 , srv1358650438:84 , srv1363215452:84 , srv1404786688:84 , srv1418563084:84 , srv1428772697:84 , srv1441230777:84 , srv1486139982:84 , srv1487458406:84 , srv1519202675:84 , srv1594453052:84 , srv1622891164:84 , srv1623524262:84 , srv162598068:84 , srv1631900979:84 , srv16393541:84 , srv1652974816:84 , srv1667650639:84 , srv1700741928:84 , srv1707768104:84 , srv1720944665:84 , srv1750898589:84 , srv1818559601:84 , srv1828301754:84 , srv1901434085:84 , srv1921155863:84 , srv1977546294:84 , srv1981746023:84 , srv2014826720:84 , srv2017674752:84 , srv2049315006:84 , srv2059309922:84 , srv206065548:84 , srv2092732436:84 , srv2111047790:84 , srv211318396:84 , srv2114117980:84 , srv2117728392:84 , srv2135325644:84 , srv2135386682:84 , srv2137873411:84 , srv226488473:84 , srv236764756:84 , srv264479876:84 , srv270394993:84 , srv324458814:84 , srv325805454:84 , srv33744095:84 , srv341716996:84 , srv34274117:84 , srv389287954:84 , srv41011186:84 , srv423962682:84 , srv425940983:84 , srv441235303:84 , srv505095366:84 , srv506930456:84 , srv554342822:84 , srv559625764:84 , srv626906301:84 , srv656185783:84 , srv667244869:84 , srv755142258:84 , srv783452215:84 , srv787217009:84 , srv824017092:84 , srv887898934:84 , srv915544959:84 , srv916491025:84 , srv930776292:84 , srv937739140:84 , srv956818730:84 , srv958395246:84 , srv104425584:85 , srv1478909148:85 , srv2106949010:85 , srv260781321:85 , srv407240095:85 , srv98143947:85 , srv1771971753:684 } 2023-03-06 12:17:28,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:28,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv626906301=84, srv104425584=1, srv2049315006=49, srv407240095=75, srv260781321=66, srv1351993416=16, srv1363215452=18, srv2011988751=46, srv2135325644=61, srv1404786688=19, srv937739140=96, srv211318396=57, srv656185783=85, srv41011186=76, srv958395246=98, srv1418563084=20, srv236764756=65, srv1298587597=11, srv425940983=78, srv1198585277=7, srv341716996=72, srv787217009=89, srv755142258=87, srv2111047790=56, srv1750898589=38, srv1070240337=3, srv2137873411=63, srv783452215=88, srv1358650438=17, srv270394993=68, srv1348267926=15, srv956818730=97, srv1622891164=28, srv2059309922=50, srv554342822=82, srv264479876=67, srv206065548=51, srv916491025=94, srv1667650639=34, srv1080827724=4, srv1921155863=43, srv2135386682=62, srv505095366=80, srv1700741928=35, srv887898934=91, srv1707768104=36, srv2119060764=60, srv1901434085=42, srv389287954=74, srv1272796615=10, srv667244869=86, srv325805454=70, srv162598068=30, srv890554055=92, srv1061548863=2, srv1345504288=14, srv1818559601=40, srv1981746023=45, srv1441230777=22, srv324458814=69, srv1486139982=24, srv2092732436=53, srv915544959=93, srv1594453052=27, srv1631900979=31, srv1319165979=13, srv1478909148=23, srv1623524262=29, srv2014826720=47, srv1828301754=41, srv1159527066=5, srv16393541=32, srv2114117980=58, srv33744095=71, srv930776292=95, srv1428772697=21, srv1487458406=25, srv2091361425=52, srv34274117=73, srv441235303=79, srv2117728392=59, srv559625764=83, srv1229761884=9, srv2017674752=48, srv1652974816=33, srv2106949010=55, srv1977546294=44, srv1771971753=39, srv1036071240=0, srv1519202675=26, srv226488473=64, srv13023610=12, srv423962682=77, srv98143947=99, srv1195190387=6, srv1720944665=37, srv2100802429=54, srv1203842989=8, srv824017092=90, srv506930456=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-06 12:17:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-06 12:17:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-06 12:17:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-06 12:17:28,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-06 12:17:28,697 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:28,697 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv890554055:82 , srv2011988751:83 , srv2091361425:83 , srv2100802429:83 , srv2119060764:83 , srv1036071240:84 , srv1061548863:84 , srv1070240337:84 , srv1080827724:84 , srv1159527066:84 , srv1195190387:84 , srv1198585277:84 , srv1203842989:84 , srv1229761884:84 , srv1272796615:84 , srv1298587597:84 , srv13023610:84 , srv1319165979:84 , srv1345504288:84 , srv1348267926:84 , srv1351993416:84 , srv1358650438:84 , srv1363215452:84 , srv1404786688:84 , srv1418563084:84 , srv1428772697:84 , srv1441230777:84 , srv1486139982:84 , srv1487458406:84 , srv1519202675:84 , srv1594453052:84 , srv1622891164:84 , srv1623524262:84 , srv162598068:84 , srv1631900979:84 , srv16393541:84 , srv1652974816:84 , srv1667650639:84 , srv1700741928:84 , srv1707768104:84 , srv1720944665:84 , srv1750898589:84 , srv1818559601:84 , srv1828301754:84 , srv1901434085:84 , srv1921155863:84 , srv1977546294:84 , srv1981746023:84 , srv2014826720:84 , srv2017674752:84 , srv2049315006:84 , srv2059309922:84 , srv206065548:84 , srv2092732436:84 , srv2111047790:84 , srv211318396:84 , srv2114117980:84 , srv2117728392:84 , srv2135325644:84 , srv2135386682:84 , srv2137873411:84 , srv226488473:84 , srv236764756:84 , srv264479876:84 , srv270394993:84 , srv324458814:84 , srv325805454:84 , srv33744095:84 , srv341716996:84 , srv34274117:84 , srv389287954:84 , srv41011186:84 , srv423962682:84 , srv425940983:84 , srv441235303:84 , srv505095366:84 , srv506930456:84 , srv554342822:84 , srv559625764:84 , srv626906301:84 , srv656185783:84 , srv667244869:84 , srv755142258:84 , srv783452215:84 , srv787217009:84 , srv824017092:84 , srv887898934:84 , srv915544959:84 , srv916491025:84 , srv930776292:84 , srv937739140:84 , srv956818730:84 , srv958395246:84 , srv104425584:85 , srv1478909148:85 , srv2106949010:85 , srv260781321:85 , srv407240095:85 , srv98143947:85 , srv1771971753:684 } 2023-03-06 12:17:28,720 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 122), ProcessCount=160 (was 160), AvailableMemoryMB=9671 (was 10169) 2023-03-06 12:17:28,727 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=122, ProcessCount=160, AvailableMemoryMB=9671 2023-03-06 12:17:28,727 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-06 12:17:28,728 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-06 12:17:28,735 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004193235:84 , srv1031301389:84 , srv1110751535:84 , srv1132895638:84 , srv1197220337:84 , srv1254940269:84 , srv1321466125:84 , srv1400363180:84 , srv1417452052:84 , srv1546151402:84 , srv1622927951:84 , srv1625956775:84 , srv171069726:84 , srv1729844307:84 , srv1738691203:84 , srv1825099005:84 , srv1892670187:84 , srv1938773576:84 , srv1952719646:84 , srv2029721873:84 , srv238693875:84 , srv333885836:84 , srv380954636:84 , srv469982072:84 , srv53318439:84 , srv549619192:84 , srv65829941:84 , srv910666048:84 , srv911875141:84 , srv216667238:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-06 12:17:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:17:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1729844307=13, srv380954636=23, srv1031301389=1, srv1400363180=7, srv469982072=24, srv1622927951=10, srv910666048=28, srv911875141=29, srv1197220337=4, srv549619192=26, srv1738691203=14, srv1132895638=3, srv1625956775=11, srv1004193235=0, srv171069726=12, srv333885836=22, srv65829941=27, srv53318439=25, srv1952719646=18, srv2029721873=19, srv1546151402=9, srv238693875=21, srv1110751535=2, srv1825099005=15, srv1321466125=6, srv1417452052=8, srv1938773576=17, srv1892670187=16, srv1254940269=5, srv216667238=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:17:28,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:17:28,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-06 12:17:28,762 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0970457893348387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:17:28,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912498831377637, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0970457893348387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-06 12:19:28,749 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6155801 different iterations. Found a solution that moves 1858 regions; Going from a computed imbalance of 0.9912498831377637 to a new imbalance of 4.336664689392977E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6881481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:19:28,756 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004193235:90 , srv1031301389:90 , srv1110751535:90 , srv1132895638:90 , srv1197220337:90 , srv1254940269:90 , srv1321466125:90 , srv1400363180:90 , srv1417452052:90 , srv1546151402:90 , srv1622927951:90 , srv1625956775:90 , srv171069726:90 , srv1729844307:90 , srv1738691203:90 , srv1825099005:90 , srv1892670187:90 , srv1938773576:90 , srv1952719646:90 , srv2029721873:90 , srv216667238:90 , srv238693875:90 , srv333885836:90 , srv380954636:90 , srv469982072:90 , srv53318439:90 , srv549619192:90 , srv65829941:90 , srv910666048:90 , srv911875141:90 } 2023-03-06 12:19:28,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-06 12:19:28,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1729844307=13, srv380954636=23, srv1031301389=1, srv1400363180=7, srv469982072=24, srv1622927951=10, srv910666048=28, srv911875141=29, srv1197220337=4, srv549619192=26, srv1738691203=14, srv1132895638=3, srv1625956775=11, srv1004193235=0, srv171069726=12, srv333885836=22, srv65829941=27, srv53318439=25, srv1952719646=18, srv2029721873=19, srv1546151402=9, srv238693875=21, srv1110751535=2, srv1825099005=15, srv1321466125=6, srv1417452052=8, srv1938773576=17, srv1892670187=16, srv1254940269=5, srv216667238=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-06 12:19:28,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-06 12:19:28,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-06 12:19:28,779 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-06 12:19:28,779 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1004193235:90 , srv1031301389:90 , srv1110751535:90 , srv1132895638:90 , srv1197220337:90 , srv1254940269:90 , srv1321466125:90 , srv1400363180:90 , srv1417452052:90 , srv1546151402:90 , srv1622927951:90 , srv1625956775:90 , srv171069726:90 , srv1729844307:90 , srv1738691203:90 , srv1825099005:90 , srv1892670187:90 , srv1938773576:90 , srv1952719646:90 , srv2029721873:90 , srv216667238:90 , srv238693875:90 , srv333885836:90 , srv380954636:90 , srv469982072:90 , srv53318439:90 , srv549619192:90 , srv65829941:90 , srv910666048:90 , srv911875141:90 } 2023-03-06 12:19:28,793 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 122) - SystemLoadAverage LEAK? -, ProcessCount=160 (was 160), AvailableMemoryMB=10066 (was 9671) - AvailableMemoryMB LEAK? -