2023-05-04 17:53:04,245 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-04 17:53:04,558 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-04 17:53:04,666 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-04 17:53:04,721 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-04 17:53:04,730 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-04 17:53:04,743 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=300, ProcessCount=175, AvailableMemoryMB=12942 2023-05-04 17:53:04,745 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-04 17:53:04,746 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-04 17:53:04,882 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019991497:84 , srv1067808777:84 , srv1086285064:84 , srv1113387883:84 , srv1156706833:84 , srv1173667820:84 , srv1180823493:84 , srv118546843:84 , srv1194803377:84 , srv1254237507:84 , srv1291253539:84 , srv130031754:84 , srv1311143682:84 , srv1316957392:84 , srv1337647351:84 , srv1338675795:84 , srv1371865362:84 , srv1387572748:84 , srv1391645887:84 , srv1410984803:84 , srv1431033286:84 , srv1442946939:84 , srv1456848280:84 , srv1471946778:84 , srv1498938963:84 , srv14998451:84 , srv1514613951:84 , srv1604869141:84 , srv160836206:84 , srv1613418657:84 , srv1614999380:84 , srv1622629250:84 , srv1653528489:84 , srv1656367943:84 , srv1663341361:84 , srv1664833900:84 , srv1696600931:84 , srv1709106342:84 , srv1769640056:84 , srv1809825747:84 , srv1819296079:84 , srv1825244780:84 , srv1855565492:84 , srv1856305420:84 , srv1884355888:84 , srv1966731838:84 , srv1969419574:84 , srv1974518592:84 , srv1990554046:84 , srv206357447:84 , srv2067420430:84 , srv2076283577:84 , srv2084795230:84 , srv213723941:84 , srv243749410:84 , srv248945043:84 , srv256535171:84 , srv283595558:84 , srv285477523:84 , srv330200560:84 , srv340432275:84 , srv369247542:84 , srv380891253:84 , srv3839031:84 , srv402767866:84 , srv410428984:84 , srv410547601:84 , srv426134113:84 , srv453293829:84 , srv461804934:84 , srv482721191:84 , srv492534725:84 , srv509926651:84 , srv523362803:84 , srv528811410:84 , srv534367956:84 , srv542322287:84 , srv559375901:84 , srv591219335:84 , srv598093800:84 , srv62442885:84 , srv641892545:84 , srv644976526:84 , srv697471395:84 , srv709021703:84 , srv713980296:84 , srv725874573:84 , srv746152750:84 , srv791676881:84 , srv796814887:84 , srv798531292:84 , srv805058819:84 , srv808825091:84 , srv829063619:84 , srv854624010:84 , srv880666552:84 , srv889601766:84 , srv92573389:84 , srv949024766:84 , srv1565879095:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-04 17:53:04,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:04,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv453293829=69, srv542322287=77, srv1809825747=40, srv1825244780=42, srv591219335=79, srv854624010=95, srv949024766=99, srv130031754=11, srv283595558=58, srv1387572748=17, srv791676881=89, srv1442946939=21, srv697471395=84, srv340432275=61, srv1622629250=32, srv509926651=73, srv1969419574=47, srv1855565492=43, srv1656367943=34, srv410547601=67, srv1086285064=2, srv160836206=29, srv1291253539=10, srv1653528489=33, srv725874573=87, srv796814887=90, srv1819296079=41, srv1019991497=0, srv1769640056=39, srv330200560=60, srv380891253=63, srv1431033286=20, srv2067420430=51, srv1410984803=19, srv805058819=92, srv829063619=94, srv285477523=59, srv889601766=97, srv1664833900=36, srv1156706833=4, srv808825091=93, srv1856305420=44, srv880666552=96, srv1337647351=14, srv1391645887=18, srv256535171=57, srv461804934=70, srv1311143682=12, srv410428984=66, srv243749410=55, srv559375901=78, srv709021703=85, srv1613418657=30, srv482721191=71, srv528811410=75, srv1604869141=28, srv369247542=62, srv1514613951=26, srv206357447=50, srv118546843=7, srv426134113=68, srv713980296=86, srv798531292=91, srv248945043=56, srv1456848280=22, srv2084795230=53, srv1371865362=16, srv1663341361=35, srv1471946778=23, srv1254237507=9, srv1565879095=27, srv1966731838=46, srv1709106342=38, srv1990554046=49, srv492534725=72, srv534367956=76, srv746152750=88, srv14998451=25, srv1316957392=13, srv1113387883=3, srv1614999380=31, srv523362803=74, srv1173667820=5, srv3839031=64, srv644976526=83, srv1498938963=24, srv62442885=81, srv1194803377=8, srv1180823493=6, srv402767866=65, srv1974518592=48, srv2076283577=52, srv1884355888=45, srv598093800=80, srv641892545=82, srv1067808777=1, srv1338675795=15, srv1696600931=37, srv92573389=98, srv213723941=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:05,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:05,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:05,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:05,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:05,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:05,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-04 17:53:05,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-04 17:53:05,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-04 17:53:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-04 17:53:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-04 17:53:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-04 17:53:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-04 17:53:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-04 17:53:05,180 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:05,181 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-04 17:53:05,181 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-04 17:53:05,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 440 ms to try 5000 different iterations. Found a solution that moves 4751 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17392046157763597. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0372539104942435, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05654075931152791, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5278888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0372539104942435, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16833333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:05,499 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv14998451:82 , srv1019991497:83 , srv1180823493:83 , srv1337647351:83 , srv1371865362:83 , srv1431033286:83 , srv1471946778:83 , srv1604869141:83 , srv1974518592:83 , srv213723941:83 , srv509926651:83 , srv641892545:83 , srv697471395:83 , srv1086285064:84 , srv1113387883:84 , srv1291253539:84 , srv1653528489:84 , srv1696600931:84 , srv1825244780:84 , srv796814887:84 , srv829063619:84 , srv1456848280:85 , srv1664833900:85 , srv1819296079:85 , srv1856305420:85 , srv1990554046:85 , srv206357447:85 , srv256535171:85 , srv283595558:85 , srv3839031:85 , srv410547601:85 , srv709021703:85 , srv92573389:85 , srv1156706833:86 , srv1173667820:86 , srv1311143682:86 , srv1663341361:86 , srv1709106342:86 , srv1769640056:86 , srv1809825747:86 , srv2076283577:86 , srv248945043:86 , srv340432275:86 , srv380891253:86 , srv482721191:86 , srv542322287:86 , srv598093800:86 , srv644976526:86 , srv725874573:86 , srv746152750:86 , srv791676881:86 , srv1194803377:87 , srv1338675795:87 , srv1387572748:87 , srv1391645887:87 , srv1442946939:87 , srv1656367943:87 , srv1966731838:87 , srv2084795230:87 , srv523362803:87 , srv528811410:87 , srv559375901:87 , srv62442885:87 , srv880666552:87 , srv1254237507:88 , srv1410984803:88 , srv1613418657:88 , srv1884355888:88 , srv330200560:88 , srv369247542:88 , srv453293829:88 , srv492534725:88 , srv854624010:88 , srv949024766:88 , srv1067808777:89 , srv130031754:89 , srv1316957392:89 , srv1498938963:89 , srv160836206:89 , srv1614999380:89 , srv1855565492:89 , srv2067420430:89 , srv243749410:89 , srv461804934:89 , srv591219335:89 , srv805058819:89 , srv1514613951:90 , srv1622629250:90 , srv1969419574:90 , srv410428984:90 , srv713980296:90 , srv889601766:90 , srv118546843:91 , srv285477523:91 , srv402767866:91 , srv426134113:91 , srv808825091:91 , srv798531292:92 , srv534367956:95 , srv1565879095:421 } 2023-05-04 17:53:05,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:05,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv453293829=69, srv542322287=77, srv1809825747=40, srv1825244780=42, srv591219335=79, srv854624010=95, srv949024766=99, srv130031754=11, srv283595558=58, srv1387572748=17, srv791676881=89, srv1442946939=21, srv697471395=84, srv340432275=61, srv1622629250=32, srv509926651=73, srv1969419574=47, srv1855565492=43, srv1656367943=34, srv410547601=67, srv1086285064=2, srv160836206=29, srv1291253539=10, srv1653528489=33, srv725874573=87, srv796814887=90, srv1819296079=41, srv1019991497=0, srv1769640056=39, srv330200560=60, srv380891253=63, srv1431033286=20, srv2067420430=51, srv1410984803=19, srv805058819=92, srv829063619=94, srv285477523=59, srv889601766=97, srv1664833900=36, srv1156706833=4, srv808825091=93, srv1856305420=44, srv880666552=96, srv1337647351=14, srv1391645887=18, srv256535171=57, srv461804934=70, srv1311143682=12, srv410428984=66, srv243749410=55, srv559375901=78, srv709021703=85, srv1613418657=30, srv482721191=71, srv528811410=75, srv1604869141=28, srv369247542=62, srv1514613951=26, srv206357447=50, srv118546843=7, srv426134113=68, srv713980296=86, srv798531292=91, srv248945043=56, srv1456848280=22, srv2084795230=53, srv1371865362=16, srv1663341361=35, srv1471946778=23, srv1254237507=9, srv1565879095=27, srv1966731838=46, srv1709106342=38, srv1990554046=49, srv492534725=72, srv534367956=76, srv746152750=88, srv14998451=25, srv1316957392=13, srv1113387883=3, srv1614999380=31, srv523362803=74, srv1173667820=5, srv3839031=64, srv644976526=83, srv1498938963=24, srv62442885=81, srv1194803377=8, srv1180823493=6, srv402767866=65, srv1974518592=48, srv2076283577=52, srv1884355888=45, srv598093800=80, srv641892545=82, srv1067808777=1, srv1338675795=15, srv1696600931=37, srv92573389=98, srv213723941=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:05,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:05,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:05,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:05,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:05,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:05,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:05,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:05,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:05,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:05,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:05,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:05,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:05,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:05,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:05,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:05,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:05,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:05,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:05,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:05,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-04 17:53:05,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-04 17:53:05,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-04 17:53:05,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-04 17:53:05,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-04 17:53:05,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-04 17:53:05,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-04 17:53:05,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-04 17:53:05,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-04 17:53:05,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-04 17:53:05,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-04 17:53:05,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-04 17:53:05,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-04 17:53:05,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-04 17:53:05,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-04 17:53:05,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-04 17:53:05,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-04 17:53:05,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-04 17:53:05,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-04 17:53:05,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-04 17:53:05,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-04 17:53:05,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-04 17:53:05,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-04 17:53:05,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-04 17:53:05,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-04 17:53:05,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-04 17:53:05,676 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0372539104942435, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05654075931152791, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0372539104942435, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16833333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:05,676 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-04 17:53:05,677 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17388719436460157, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0372539104942435, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05654075931152791, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0372539104942435, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16833333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-04 17:53:05,869 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 289 ms to try 5000 different iterations. Found a solution that moves 2617 regions; Going from a computed imbalance of 0.17388719436460157 to a new imbalance of 0.015115189749186632. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020928086059956226); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03976925590109838, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2907777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020928086059956226); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:05,877 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv534367956:78 , srv402767866:79 , srv410428984:80 , srv285477523:81 , srv798531292:81 , srv889601766:81 , srv1067808777:82 , srv118546843:82 , srv1254237507:82 , srv1514613951:82 , srv1969419574:82 , srv2067420430:82 , srv492534725:82 , srv559375901:82 , srv808825091:82 , srv1316957392:83 , srv1410984803:83 , srv1855565492:83 , srv243749410:83 , srv426134113:83 , srv461804934:83 , srv713980296:83 , srv854624010:83 , srv1194803377:84 , srv130031754:84 , srv160836206:84 , srv1614999380:84 , srv1663341361:84 , srv1884355888:84 , srv330200560:84 , srv380891253:84 , srv453293829:84 , srv523362803:84 , srv528811410:84 , srv591219335:84 , srv62442885:84 , srv1387572748:85 , srv1391645887:85 , srv1622629250:85 , srv1656367943:85 , srv1769640056:85 , srv1966731838:85 , srv2084795230:85 , srv283595558:85 , srv369247542:85 , srv410547601:85 , srv482721191:85 , srv542322287:85 , srv598093800:85 , srv805058819:85 , srv949024766:85 , srv1442946939:86 , srv1498938963:86 , srv1613418657:86 , srv1653528489:86 , srv1809825747:86 , srv1819296079:86 , srv1825244780:86 , srv248945043:86 , srv256535171:86 , srv340432275:86 , srv725874573:86 , srv791676881:86 , srv1156706833:87 , srv1291253539:87 , srv1338675795:87 , srv1856305420:87 , srv206357447:87 , srv644976526:87 , srv746152750:87 , srv1019991497:88 , srv1086285064:88 , srv1113387883:88 , srv1180823493:88 , srv1311143682:88 , srv1371865362:88 , srv1456848280:88 , srv1604869141:88 , srv1664833900:88 , srv1696600931:88 , srv1709106342:88 , srv2076283577:88 , srv3839031:88 , srv697471395:88 , srv796814887:88 , srv880666552:88 , srv92573389:88 , srv1173667820:89 , srv1337647351:89 , srv1431033286:89 , srv14998451:89 , srv1974518592:89 , srv1990554046:89 , srv709021703:89 , srv829063619:89 , srv1471946778:90 , srv213723941:90 , srv509926651:91 , srv641892545:92 , srv1565879095:539 } 2023-05-04 17:53:05,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:05,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv453293829=69, srv542322287=77, srv1809825747=40, srv1825244780=42, srv591219335=79, srv854624010=95, srv949024766=99, srv130031754=11, srv283595558=58, srv1387572748=17, srv791676881=89, srv1442946939=21, srv697471395=84, srv340432275=61, srv1622629250=32, srv509926651=73, srv1969419574=47, srv1855565492=43, srv1656367943=34, srv410547601=67, srv1086285064=2, srv160836206=29, srv1291253539=10, srv1653528489=33, srv725874573=87, srv796814887=90, srv1819296079=41, srv1019991497=0, srv1769640056=39, srv330200560=60, srv380891253=63, srv1431033286=20, srv2067420430=51, srv1410984803=19, srv805058819=92, srv829063619=94, srv285477523=59, srv889601766=97, srv1664833900=36, srv1156706833=4, srv808825091=93, srv1856305420=44, srv880666552=96, srv1337647351=14, srv1391645887=18, srv256535171=57, srv461804934=70, srv1311143682=12, srv410428984=66, srv243749410=55, srv559375901=78, srv709021703=85, srv1613418657=30, srv482721191=71, srv528811410=75, srv1604869141=28, srv369247542=62, srv1514613951=26, srv206357447=50, srv118546843=7, srv426134113=68, srv713980296=86, srv798531292=91, srv248945043=56, srv1456848280=22, srv2084795230=53, srv1371865362=16, srv1663341361=35, srv1471946778=23, srv1254237507=9, srv1565879095=27, srv1966731838=46, srv1709106342=38, srv1990554046=49, srv492534725=72, srv534367956=76, srv746152750=88, srv14998451=25, srv1316957392=13, srv1113387883=3, srv1614999380=31, srv523362803=74, srv1173667820=5, srv3839031=64, srv644976526=83, srv1498938963=24, srv62442885=81, srv1194803377=8, srv1180823493=6, srv402767866=65, srv1974518592=48, srv2076283577=52, srv1884355888=45, srv598093800=80, srv641892545=82, srv1067808777=1, srv1338675795=15, srv1696600931=37, srv92573389=98, srv213723941=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:05,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:05,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-04 17:53:05,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-04 17:53:05,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-04 17:53:05,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-04 17:53:05,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-04 17:53:05,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-04 17:53:05,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-04 17:53:05,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-04 17:53:05,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-04 17:53:05,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-04 17:53:05,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020928086059956226); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03976925590109838, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020928086059956226); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:05,986 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-04 17:53:05,987 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015096865123526556, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020928086059956226); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03976925590109838, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020928086059956226); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-04 17:53:06,152 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 237 ms to try 5000 different iterations. Found a solution that moves 993 regions; Going from a computed imbalance of 0.015096865123526556 to a new imbalance of 5.421952132612536E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006106262743597928); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009039134271190744); MoveCostFunction : (multiplier=7.0, imbalance=0.11033333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006106262743597928); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:06,154 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv509926651:82 , srv1173667820:83 , srv1498938963:83 , srv1622629250:83 , srv1709106342:83 , srv1990554046:83 , srv2076283577:83 , srv641892545:83 , srv880666552:83 , srv1311143682:84 , srv1338675795:84 , srv1456848280:84 , srv1471946778:84 , srv1614999380:84 , srv1664833900:84 , srv369247542:84 , srv3839031:84 , srv426134113:84 , srv591219335:84 , srv644976526:84 , srv709021703:84 , srv713980296:84 , srv746152750:84 , srv808825091:84 , srv949024766:84 , srv1019991497:85 , srv1086285064:85 , srv1113387883:85 , srv118546843:85 , srv130031754:85 , srv1316957392:85 , srv1337647351:85 , srv1387572748:85 , srv1391645887:85 , srv1431033286:85 , srv1442946939:85 , srv1514613951:85 , srv160836206:85 , srv1613418657:85 , srv1656367943:85 , srv1856305420:85 , srv1884355888:85 , srv1969419574:85 , srv1974518592:85 , srv213723941:85 , srv243749410:85 , srv248945043:85 , srv285477523:85 , srv340432275:85 , srv461804934:85 , srv534367956:85 , srv796814887:85 , srv798531292:85 , srv805058819:85 , srv829063619:85 , srv1156706833:86 , srv1180823493:86 , srv1371865362:86 , srv1410984803:86 , srv1604869141:86 , srv1696600931:86 , srv1769640056:86 , srv1809825747:86 , srv1855565492:86 , srv1966731838:86 , srv206357447:86 , srv2084795230:86 , srv256535171:86 , srv410547601:86 , srv453293829:86 , srv523362803:86 , srv528811410:86 , srv542322287:86 , srv62442885:86 , srv697471395:86 , srv725874573:86 , srv791676881:86 , srv854624010:86 , srv889601766:86 , srv92573389:86 , srv1067808777:87 , srv1194803377:87 , srv1254237507:87 , srv1291253539:87 , srv14998451:87 , srv1653528489:87 , srv1819296079:87 , srv1825244780:87 , srv2067420430:87 , srv283595558:87 , srv330200560:87 , srv380891253:87 , srv482721191:87 , srv559375901:87 , srv1663341361:88 , srv402767866:88 , srv410428984:88 , srv492534725:88 , srv598093800:88 , srv1565879095:552 } 2023-05-04 17:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv453293829=69, srv542322287=77, srv1809825747=40, srv1825244780=42, srv591219335=79, srv854624010=95, srv949024766=99, srv130031754=11, srv283595558=58, srv1387572748=17, srv791676881=89, srv1442946939=21, srv697471395=84, srv340432275=61, srv1622629250=32, srv509926651=73, srv1969419574=47, srv1855565492=43, srv1656367943=34, srv410547601=67, srv1086285064=2, srv160836206=29, srv1291253539=10, srv1653528489=33, srv725874573=87, srv796814887=90, srv1819296079=41, srv1019991497=0, srv1769640056=39, srv330200560=60, srv380891253=63, srv1431033286=20, srv2067420430=51, srv1410984803=19, srv805058819=92, srv829063619=94, srv285477523=59, srv889601766=97, srv1664833900=36, srv1156706833=4, srv808825091=93, srv1856305420=44, srv880666552=96, srv1337647351=14, srv1391645887=18, srv256535171=57, srv461804934=70, srv1311143682=12, srv410428984=66, srv243749410=55, srv559375901=78, srv709021703=85, srv1613418657=30, srv482721191=71, srv528811410=75, srv1604869141=28, srv369247542=62, srv1514613951=26, srv206357447=50, srv118546843=7, srv426134113=68, srv713980296=86, srv798531292=91, srv248945043=56, srv1456848280=22, srv2084795230=53, srv1371865362=16, srv1663341361=35, srv1471946778=23, srv1254237507=9, srv1565879095=27, srv1966731838=46, srv1709106342=38, srv1990554046=49, srv492534725=72, srv534367956=76, srv746152750=88, srv14998451=25, srv1316957392=13, srv1113387883=3, srv1614999380=31, srv523362803=74, srv1173667820=5, srv3839031=64, srv644976526=83, srv1498938963=24, srv62442885=81, srv1194803377=8, srv1180823493=6, srv402767866=65, srv1974518592=48, srv2076283577=52, srv1884355888=45, srv598093800=80, srv641892545=82, srv1067808777=1, srv1338675795=15, srv1696600931=37, srv92573389=98, srv213723941=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-04 17:53:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-04 17:53:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-04 17:53:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-04 17:53:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-04 17:53:06,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-04 17:53:06,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006106262743597928); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009039134271190744); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006106262743597928); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:06,262 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-04 17:53:06,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.352420786489275E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006106262743597928); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009039134271190744); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006106262743597928); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-04 17:53:06,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations. Found a solution that moves 271 regions; Going from a computed imbalance of 5.352420786489275E-4 to a new imbalance of 2.9639876490900853E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.030111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:06,538 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1613418657:83 , srv330200560:83 , srv598093800:83 , srv725874573:83 , srv805058819:83 , srv1067808777:84 , srv1113387883:84 , srv1156706833:84 , srv1173667820:84 , srv118546843:84 , srv1194803377:84 , srv1291253539:84 , srv130031754:84 , srv1311143682:84 , srv1338675795:84 , srv1410984803:84 , srv1442946939:84 , srv14998451:84 , srv160836206:84 , srv1622629250:84 , srv1663341361:84 , srv1696600931:84 , srv1809825747:84 , srv1819296079:84 , srv1855565492:84 , srv1966731838:84 , srv206357447:84 , srv2067420430:84 , srv2084795230:84 , srv213723941:84 , srv380891253:84 , srv410428984:84 , srv426134113:84 , srv453293829:84 , srv482721191:84 , srv492534725:84 , srv641892545:84 , srv709021703:84 , srv791676881:84 , srv829063619:84 , srv880666552:84 , srv92573389:84 , srv1086285064:85 , srv1180823493:85 , srv1254237507:85 , srv1316957392:85 , srv1337647351:85 , srv1371865362:85 , srv1387572748:85 , srv1391645887:85 , srv1431033286:85 , srv1456848280:85 , srv1471946778:85 , srv1498938963:85 , srv1514613951:85 , srv1604869141:85 , srv1614999380:85 , srv1653528489:85 , srv1656367943:85 , srv1664833900:85 , srv1709106342:85 , srv1769640056:85 , srv1825244780:85 , srv1856305420:85 , srv1884355888:85 , srv1969419574:85 , srv1974518592:85 , srv1990554046:85 , srv2076283577:85 , srv243749410:85 , srv248945043:85 , srv256535171:85 , srv283595558:85 , srv285477523:85 , srv340432275:85 , srv369247542:85 , srv3839031:85 , srv402767866:85 , srv410547601:85 , srv461804934:85 , srv523362803:85 , srv528811410:85 , srv534367956:85 , srv542322287:85 , srv559375901:85 , srv591219335:85 , srv62442885:85 , srv644976526:85 , srv697471395:85 , srv713980296:85 , srv746152750:85 , srv796814887:85 , srv798531292:85 , srv808825091:85 , srv854624010:85 , srv889601766:85 , srv949024766:85 , srv1019991497:86 , srv509926651:86 , srv1565879095:630 } 2023-05-04 17:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv453293829=69, srv542322287=77, srv1809825747=40, srv1825244780=42, srv591219335=79, srv854624010=95, srv949024766=99, srv130031754=11, srv283595558=58, srv1387572748=17, srv791676881=89, srv1442946939=21, srv697471395=84, srv340432275=61, srv1622629250=32, srv509926651=73, srv1969419574=47, srv1855565492=43, srv1656367943=34, srv410547601=67, srv1086285064=2, srv160836206=29, srv1291253539=10, srv1653528489=33, srv725874573=87, srv796814887=90, srv1819296079=41, srv1019991497=0, srv1769640056=39, srv330200560=60, srv380891253=63, srv1431033286=20, srv2067420430=51, srv1410984803=19, srv805058819=92, srv829063619=94, srv285477523=59, srv889601766=97, srv1664833900=36, srv1156706833=4, srv808825091=93, srv1856305420=44, srv880666552=96, srv1337647351=14, srv1391645887=18, srv256535171=57, srv461804934=70, srv1311143682=12, srv410428984=66, srv243749410=55, srv559375901=78, srv709021703=85, srv1613418657=30, srv482721191=71, srv528811410=75, srv1604869141=28, srv369247542=62, srv1514613951=26, srv206357447=50, srv118546843=7, srv426134113=68, srv713980296=86, srv798531292=91, srv248945043=56, srv1456848280=22, srv2084795230=53, srv1371865362=16, srv1663341361=35, srv1471946778=23, srv1254237507=9, srv1565879095=27, srv1966731838=46, srv1709106342=38, srv1990554046=49, srv492534725=72, srv534367956=76, srv746152750=88, srv14998451=25, srv1316957392=13, srv1113387883=3, srv1614999380=31, srv523362803=74, srv1173667820=5, srv3839031=64, srv644976526=83, srv1498938963=24, srv62442885=81, srv1194803377=8, srv1180823493=6, srv402767866=65, srv1974518592=48, srv2076283577=52, srv1884355888=45, srv598093800=80, srv641892545=82, srv1067808777=1, srv1338675795=15, srv1696600931=37, srv92573389=98, srv213723941=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:06,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:06,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:06,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-04 17:53:06,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-04 17:53:06,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-04 17:53:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-04 17:53:06,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-04 17:53:06,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-04 17:53:06,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-04 17:53:06,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-04 17:53:06,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-04 17:53:06,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-04 17:53:06,661 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:06,661 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-04 17:53:06,661 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.7742293933055592E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-04 17:53:06,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 253 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 2.7742293933055592E-5 to a new imbalance of 7.906488439809095E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:06,828 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1156706833:83 , srv1498938963:83 , srv402767866:83 , srv534367956:83 , srv798531292:83 , srv92573389:83 , srv1019991497:84 , srv1067808777:84 , srv1086285064:84 , srv1173667820:84 , srv1180823493:84 , srv118546843:84 , srv1194803377:84 , srv1254237507:84 , srv1291253539:84 , srv130031754:84 , srv1311143682:84 , srv1316957392:84 , srv1337647351:84 , srv1338675795:84 , srv1371865362:84 , srv1387572748:84 , srv1391645887:84 , srv1431033286:84 , srv1442946939:84 , srv1456848280:84 , srv1471946778:84 , srv14998451:84 , srv1514613951:84 , srv1604869141:84 , srv160836206:84 , srv1613418657:84 , srv1614999380:84 , srv1622629250:84 , srv1653528489:84 , srv1656367943:84 , srv1663341361:84 , srv1664833900:84 , srv1696600931:84 , srv1709106342:84 , srv1769640056:84 , srv1809825747:84 , srv1819296079:84 , srv1825244780:84 , srv1855565492:84 , srv1856305420:84 , srv1884355888:84 , srv1966731838:84 , srv1969419574:84 , srv1974518592:84 , srv1990554046:84 , srv206357447:84 , srv2067420430:84 , srv2076283577:84 , srv2084795230:84 , srv213723941:84 , srv243749410:84 , srv248945043:84 , srv256535171:84 , srv283595558:84 , srv285477523:84 , srv330200560:84 , srv340432275:84 , srv369247542:84 , srv3839031:84 , srv410428984:84 , srv426134113:84 , srv453293829:84 , srv461804934:84 , srv482721191:84 , srv492534725:84 , srv509926651:84 , srv523362803:84 , srv528811410:84 , srv542322287:84 , srv591219335:84 , srv598093800:84 , srv62442885:84 , srv641892545:84 , srv644976526:84 , srv697471395:84 , srv709021703:84 , srv713980296:84 , srv746152750:84 , srv791676881:84 , srv796814887:84 , srv805058819:84 , srv808825091:84 , srv829063619:84 , srv854624010:84 , srv880666552:84 , srv889601766:84 , srv949024766:84 , srv1113387883:85 , srv1410984803:85 , srv380891253:85 , srv410547601:85 , srv559375901:85 , srv725874573:85 , srv1565879095:684 } 2023-05-04 17:53:06,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:06,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv453293829=69, srv542322287=77, srv1809825747=40, srv1825244780=42, srv591219335=79, srv854624010=95, srv949024766=99, srv130031754=11, srv283595558=58, srv1387572748=17, srv791676881=89, srv1442946939=21, srv697471395=84, srv340432275=61, srv1622629250=32, srv509926651=73, srv1969419574=47, srv1855565492=43, srv1656367943=34, srv410547601=67, srv1086285064=2, srv160836206=29, srv1291253539=10, srv1653528489=33, srv725874573=87, srv796814887=90, srv1819296079=41, srv1019991497=0, srv1769640056=39, srv330200560=60, srv380891253=63, srv1431033286=20, srv2067420430=51, srv1410984803=19, srv805058819=92, srv829063619=94, srv285477523=59, srv889601766=97, srv1664833900=36, srv1156706833=4, srv808825091=93, srv1856305420=44, srv880666552=96, srv1337647351=14, srv1391645887=18, srv256535171=57, srv461804934=70, srv1311143682=12, srv410428984=66, srv243749410=55, srv559375901=78, srv709021703=85, srv1613418657=30, srv482721191=71, srv528811410=75, srv1604869141=28, srv369247542=62, srv1514613951=26, srv206357447=50, srv118546843=7, srv426134113=68, srv713980296=86, srv798531292=91, srv248945043=56, srv1456848280=22, srv2084795230=53, srv1371865362=16, srv1663341361=35, srv1471946778=23, srv1254237507=9, srv1565879095=27, srv1966731838=46, srv1709106342=38, srv1990554046=49, srv492534725=72, srv534367956=76, srv746152750=88, srv14998451=25, srv1316957392=13, srv1113387883=3, srv1614999380=31, srv523362803=74, srv1173667820=5, srv3839031=64, srv644976526=83, srv1498938963=24, srv62442885=81, srv1194803377=8, srv1180823493=6, srv402767866=65, srv1974518592=48, srv2076283577=52, srv1884355888=45, srv598093800=80, srv641892545=82, srv1067808777=1, srv1338675795=15, srv1696600931=37, srv92573389=98, srv213723941=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:06,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-04 17:53:06,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-04 17:53:06,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-04 17:53:06,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-04 17:53:06,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-04 17:53:06,920 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:06,921 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1156706833:83 , srv1498938963:83 , srv402767866:83 , srv534367956:83 , srv798531292:83 , srv92573389:83 , srv1019991497:84 , srv1067808777:84 , srv1086285064:84 , srv1173667820:84 , srv1180823493:84 , srv118546843:84 , srv1194803377:84 , srv1254237507:84 , srv1291253539:84 , srv130031754:84 , srv1311143682:84 , srv1316957392:84 , srv1337647351:84 , srv1338675795:84 , srv1371865362:84 , srv1387572748:84 , srv1391645887:84 , srv1431033286:84 , srv1442946939:84 , srv1456848280:84 , srv1471946778:84 , srv14998451:84 , srv1514613951:84 , srv1604869141:84 , srv160836206:84 , srv1613418657:84 , srv1614999380:84 , srv1622629250:84 , srv1653528489:84 , srv1656367943:84 , srv1663341361:84 , srv1664833900:84 , srv1696600931:84 , srv1709106342:84 , srv1769640056:84 , srv1809825747:84 , srv1819296079:84 , srv1825244780:84 , srv1855565492:84 , srv1856305420:84 , srv1884355888:84 , srv1966731838:84 , srv1969419574:84 , srv1974518592:84 , srv1990554046:84 , srv206357447:84 , srv2067420430:84 , srv2076283577:84 , srv2084795230:84 , srv213723941:84 , srv243749410:84 , srv248945043:84 , srv256535171:84 , srv283595558:84 , srv285477523:84 , srv330200560:84 , srv340432275:84 , srv369247542:84 , srv3839031:84 , srv410428984:84 , srv426134113:84 , srv453293829:84 , srv461804934:84 , srv482721191:84 , srv492534725:84 , srv509926651:84 , srv523362803:84 , srv528811410:84 , srv542322287:84 , srv591219335:84 , srv598093800:84 , srv62442885:84 , srv641892545:84 , srv644976526:84 , srv697471395:84 , srv709021703:84 , srv713980296:84 , srv746152750:84 , srv791676881:84 , srv796814887:84 , srv805058819:84 , srv808825091:84 , srv829063619:84 , srv854624010:84 , srv880666552:84 , srv889601766:84 , srv949024766:84 , srv1113387883:85 , srv1410984803:85 , srv380891253:85 , srv410547601:85 , srv559375901:85 , srv725874573:85 , srv1565879095:684 } 2023-05-04 17:53:06,945 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=300 (was 300), ProcessCount=175 (was 175), AvailableMemoryMB=12310 (was 12942) 2023-05-04 17:53:06,951 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=300, ProcessCount=175, AvailableMemoryMB=12310 2023-05-04 17:53:06,952 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-04 17:53:06,952 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-04 17:53:06,963 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv114597:84 , srv1209087826:84 , srv1213241148:84 , srv1220063352:84 , srv1341660836:84 , srv1351444264:84 , srv1386664215:84 , srv139719745:84 , srv1407070866:84 , srv1518508510:84 , srv1545581782:84 , srv1853420727:84 , srv1865324428:84 , srv1932039093:84 , srv1962107372:84 , srv2017249837:84 , srv2023592335:84 , srv2031876592:84 , srv2085383048:84 , srv2104926605:84 , srv2108990962:84 , srv262823018:84 , srv376321912:84 , srv386773439:84 , srv514536337:84 , srv65077837:84 , srv755042494:84 , srv821684923:84 , srv885344693:84 , srv1237757198:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-04 17:53:06,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:53:06,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2031876592=18, srv1351444264=6, srv885344693=29, srv2108990962=21, srv2017249837=16, srv376321912=23, srv1237757198=4, srv1341660836=5, srv2085383048=19, srv1932039093=14, srv514536337=25, srv1386664215=7, srv139719745=8, srv1213241148=2, srv386773439=24, srv65077837=26, srv2023592335=17, srv262823018=22, srv1209087826=1, srv1865324428=13, srv1962107372=15, srv1220063352=3, srv2104926605=20, srv1518508510=10, srv755042494=27, srv1407070866=9, srv114597=0, srv1853420727=12, srv821684923=28, srv1545581782=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:53:06,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:53:06,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:53:06,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:53:06,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:53:06,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:53:06,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:53:06,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:53:06,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:53:06,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:53:06,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-04 17:53:07,005 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1263606703424625, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:53:07,005 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912591201610473, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1263606703424625, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-04 17:55:06,983 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9423339 different iterations. Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912591201610473 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:55:06,989 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv114597:90 , srv1209087826:90 , srv1213241148:90 , srv1220063352:90 , srv1237757198:90 , srv1341660836:90 , srv1351444264:90 , srv1386664215:90 , srv139719745:90 , srv1407070866:90 , srv1518508510:90 , srv1545581782:90 , srv1853420727:90 , srv1865324428:90 , srv1932039093:90 , srv1962107372:90 , srv2017249837:90 , srv2023592335:90 , srv2031876592:90 , srv2085383048:90 , srv2104926605:90 , srv2108990962:90 , srv262823018:90 , srv376321912:90 , srv386773439:90 , srv514536337:90 , srv65077837:90 , srv755042494:90 , srv821684923:90 , srv885344693:90 } 2023-05-04 17:55:06,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-04 17:55:06,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2031876592=18, srv1351444264=6, srv885344693=29, srv2108990962=21, srv2017249837=16, srv376321912=23, srv1237757198=4, srv1341660836=5, srv2085383048=19, srv1932039093=14, srv514536337=25, srv1386664215=7, srv139719745=8, srv1213241148=2, srv386773439=24, srv65077837=26, srv2023592335=17, srv262823018=22, srv1209087826=1, srv1865324428=13, srv1962107372=15, srv1220063352=3, srv2104926605=20, srv1518508510=10, srv755042494=27, srv1407070866=9, srv114597=0, srv1853420727=12, srv821684923=28, srv1545581782=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-04 17:55:06,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-04 17:55:06,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-04 17:55:06,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-04 17:55:06,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-04 17:55:06,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-04 17:55:06,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-04 17:55:06,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-04 17:55:06,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-04 17:55:06,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-04 17:55:06,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-04 17:55:07,005 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-04 17:55:07,005 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv114597:90 , srv1209087826:90 , srv1213241148:90 , srv1220063352:90 , srv1237757198:90 , srv1341660836:90 , srv1351444264:90 , srv1386664215:90 , srv139719745:90 , srv1407070866:90 , srv1518508510:90 , srv1545581782:90 , srv1853420727:90 , srv1865324428:90 , srv1932039093:90 , srv1962107372:90 , srv2017249837:90 , srv2023592335:90 , srv2031876592:90 , srv2085383048:90 , srv2104926605:90 , srv2108990962:90 , srv262823018:90 , srv376321912:90 , srv386773439:90 , srv514536337:90 , srv65077837:90 , srv755042494:90 , srv821684923:90 , srv885344693:90 } 2023-05-04 17:55:07,015 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=214 (was 300), ProcessCount=174 (was 175), AvailableMemoryMB=11411 (was 12310)