2023-10-24 07:54:47,774 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-24 07:54:48,135 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-24 07:54:48,247 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-24 07:54:48,320 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-24 07:54:48,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-24 07:54:48,343 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=170, AvailableMemoryMB=11178 2023-10-24 07:54:48,346 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-24 07:54:48,346 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-24 07:54:48,464 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009017103:84 , srv1009037560:84 , srv1023108534:84 , srv104349979:84 , srv1075423230:84 , srv1107416503:84 , srv1109186877:84 , srv1167273507:84 , srv1194503995:84 , srv1278814494:84 , srv1314998931:84 , srv1353198444:84 , srv1353494194:84 , srv1380503240:84 , srv1404385260:84 , srv1459778331:84 , srv1463708527:84 , srv1481446541:84 , srv1505012112:84 , srv1547679274:84 , srv1565709319:84 , srv1579415360:84 , srv1580011922:84 , srv158202948:84 , srv158347759:84 , srv160915758:84 , srv1615119545:84 , srv1621184483:84 , srv1686211774:84 , srv1703894567:84 , srv1717989968:84 , srv1740068716:84 , srv1766145228:84 , srv1773490758:84 , srv1786151496:84 , srv1795908067:84 , srv1824657958:84 , srv1830881713:84 , srv1838793566:84 , srv1842450759:84 , srv1849633328:84 , srv1853554412:84 , srv1874659565:84 , srv1877694213:84 , srv1883140587:84 , srv1884940367:84 , srv1909668126:84 , srv1920773618:84 , srv1937334152:84 , srv1947474154:84 , srv2024553986:84 , srv2031585610:84 , srv2047280228:84 , srv2055629832:84 , srv2069095423:84 , srv2119918665:84 , srv2127694272:84 , srv2198996:84 , srv257484949:84 , srv283087146:84 , srv284634369:84 , srv293337065:84 , srv319563501:84 , srv324470022:84 , srv365235180:84 , srv38903024:84 , srv437163549:84 , srv449196240:84 , srv450545153:84 , srv457603202:84 , srv465043560:84 , srv507617960:84 , srv509414127:84 , srv522784977:84 , srv53127904:84 , srv548895116:84 , srv572617613:84 , srv582167989:84 , srv608785619:84 , srv621387948:84 , srv627570515:84 , srv638931588:84 , srv654865882:84 , srv672471253:84 , srv70009736:84 , srv714843958:84 , srv715537649:84 , srv743105488:84 , srv815835816:84 , srv817754264:84 , srv818564504:84 , srv84340461:84 , srv855388833:84 , srv874973751:84 , srv90803343:84 , srv911531193:84 , srv967630840:84 , srv988571947:84 , srv999250868:84 , srv1276839472:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-24 07:54:48,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:48,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1877694213=44, srv1909668126=47, srv1009017103=0, srv1314998931=11, srv1920773618=48, srv1937334152=49, srv672471253=84, srv714843958=86, srv818564504=91, srv911531193=96, srv1380503240=14, srv1874659565=43, srv1838793566=39, srv90803343=95, srv1615119545=27, srv582167989=78, srv1824657958=37, srv284634369=61, srv1795908067=36, srv1947474154=50, srv70009736=85, srv522784977=74, srv1459778331=16, srv450545153=69, srv509414127=73, srv1740068716=32, srv257484949=59, srv465043560=71, srv1353494194=13, srv638931588=82, srv1579415360=22, srv2055629832=54, srv967630840=97, srv1884940367=46, srv160915758=26, srv158202948=24, srv1278814494=10, srv548895116=76, srv84340461=92, srv1773490758=34, srv1353198444=12, srv158347759=25, srv1849633328=41, srv365235180=65, srv507617960=72, srv2024553986=51, srv815835816=89, srv449196240=68, srv1547679274=20, srv817754264=90, srv1580011922=23, srv2047280228=53, srv988571947=98, srv1276839472=9, srv104349979=3, srv1842450759=40, srv324470022=64, srv1883140587=45, srv608785619=79, srv293337065=62, srv874973751=94, srv1766145228=33, srv1703894567=30, srv38903024=66, srv1009037560=1, srv283087146=60, srv1565709319=21, srv654865882=83, srv1717989968=31, srv2127694272=57, srv2119918665=56, srv319563501=63, srv1109186877=6, srv999250868=99, srv572617613=77, srv627570515=81, srv457603202=70, srv1075423230=4, srv1621184483=28, srv1194503995=8, srv1404385260=15, srv1686211774=29, srv1481446541=18, srv715537649=87, srv2069095423=55, srv621387948=80, srv1107416503=5, srv1853554412=42, srv1830881713=38, srv1786151496=35, srv437163549=67, srv2031585610=52, srv1023108534=2, srv855388833=93, srv1505012112=19, srv53127904=75, srv2198996=58, srv743105488=88, srv1167273507=7, srv1463708527=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:48,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:48,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:48,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:48,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:48,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:48,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:48,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:48,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:48,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:48,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:48,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:48,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:48,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:48,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:48,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:48,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:48,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:48,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:48,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:48,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:48,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:48,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:48,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:48,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:48,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:48,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:48,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:48,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:48,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:48,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:48,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 07:54:48,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 07:54:48,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 07:54:48,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 07:54:48,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 07:54:48,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 07:54:48,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 07:54:48,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 07:54:48,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 07:54:48,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 07:54:48,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 07:54:48,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 07:54:48,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 07:54:48,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 07:54:48,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 07:54:48,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 07:54:48,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 07:54:48,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 07:54:48,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 07:54:48,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 07:54:48,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 07:54:48,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 07:54:48,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 07:54:48,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 07:54:48,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 07:54:48,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 07:54:48,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 07:54:48,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 07:54:48,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 07:54:48,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 07:54:48,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 07:54:48,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 07:54:48,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 07:54:48,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 07:54:48,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 07:54:48,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 07:54:48,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 07:54:48,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 07:54:48,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 07:54:48,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 07:54:48,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 07:54:48,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 07:54:48,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 07:54:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 07:54:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 07:54:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 07:54:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 07:54:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 07:54:48,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 07:54:48,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 07:54:48,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 07:54:48,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 07:54:48,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 07:54:48,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 07:54:48,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 07:54:48,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 07:54:48,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 07:54:48,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 07:54:48,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 07:54:48,881 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:48,882 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 07:54:48,882 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 07:54:49,294 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 696 ms to try 5000 different iterations. Found a solution that moves 4664 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18294130998818162. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040176378808217825, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05589994267859546, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5182222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040176378808217825, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:49,319 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1075423230:81 , srv1167273507:81 , srv90803343:82 , srv1314998931:83 , srv1404385260:83 , srv1838793566:83 , srv1877694213:83 , srv1947474154:83 , srv2024553986:83 , srv2055629832:83 , srv608785619:83 , srv999250868:83 , srv1009017103:84 , srv1023108534:84 , srv1353494194:84 , srv1615119545:84 , srv1717989968:84 , srv1786151496:84 , srv2031585610:84 , srv2069095423:84 , srv2198996:84 , srv53127904:84 , srv818564504:84 , srv1481446541:85 , srv1505012112:85 , srv1547679274:85 , srv1849633328:85 , srv1853554412:85 , srv1883140587:85 , srv284634369:85 , srv324470022:85 , srv450545153:85 , srv507617960:85 , srv522784977:85 , srv582167989:85 , srv621387948:85 , srv672471253:85 , srv84340461:85 , srv874973751:85 , srv1009037560:86 , srv1107416503:86 , srv1194503995:86 , srv1278814494:86 , srv1459778331:86 , srv158202948:86 , srv1773490758:86 , srv1795908067:86 , srv1830881713:86 , srv1842450759:86 , srv1909668126:86 , srv1937334152:86 , srv2047280228:86 , srv365235180:86 , srv714843958:86 , srv911531193:86 , srv104349979:87 , srv1109186877:87 , srv1565709319:87 , srv1579415360:87 , srv160915758:87 , srv1621184483:87 , srv1703894567:87 , srv1824657958:87 , srv1884940367:87 , srv38903024:87 , srv465043560:87 , srv548895116:87 , srv627570515:87 , srv638931588:87 , srv815835816:87 , srv988571947:87 , srv1353198444:88 , srv158347759:88 , srv1740068716:88 , srv1874659565:88 , srv257484949:88 , srv283087146:88 , srv654865882:88 , srv743105488:88 , srv817754264:88 , srv967630840:88 , srv1580011922:89 , srv1686211774:89 , srv319563501:89 , srv437163549:89 , srv449196240:89 , srv509414127:89 , srv70009736:89 , srv2119918665:90 , srv293337065:90 , srv457603202:90 , srv715537649:90 , srv1463708527:91 , srv1920773618:91 , srv572617613:91 , srv1380503240:92 , srv1766145228:92 , srv855388833:92 , srv2127694272:98 , srv1276839472:447 } 2023-10-24 07:54:49,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:49,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1877694213=44, srv1909668126=47, srv1009017103=0, srv1314998931=11, srv1920773618=48, srv1937334152=49, srv672471253=84, srv714843958=86, srv818564504=91, srv911531193=96, srv1380503240=14, srv1874659565=43, srv1838793566=39, srv90803343=95, srv1615119545=27, srv582167989=78, srv1824657958=37, srv284634369=61, srv1795908067=36, srv1947474154=50, srv70009736=85, srv522784977=74, srv1459778331=16, srv450545153=69, srv509414127=73, srv1740068716=32, srv257484949=59, srv465043560=71, srv1353494194=13, srv638931588=82, srv1579415360=22, srv2055629832=54, srv967630840=97, srv1884940367=46, srv160915758=26, srv158202948=24, srv1278814494=10, srv548895116=76, srv84340461=92, srv1773490758=34, srv1353198444=12, srv158347759=25, srv1849633328=41, srv365235180=65, srv507617960=72, srv2024553986=51, srv815835816=89, srv449196240=68, srv1547679274=20, srv817754264=90, srv1580011922=23, srv2047280228=53, srv988571947=98, srv1276839472=9, srv104349979=3, srv1842450759=40, srv324470022=64, srv1883140587=45, srv608785619=79, srv293337065=62, srv874973751=94, srv1766145228=33, srv1703894567=30, srv38903024=66, srv1009037560=1, srv283087146=60, srv1565709319=21, srv654865882=83, srv1717989968=31, srv2127694272=57, srv2119918665=56, srv319563501=63, srv1109186877=6, srv999250868=99, srv572617613=77, srv627570515=81, srv457603202=70, srv1075423230=4, srv1621184483=28, srv1194503995=8, srv1404385260=15, srv1686211774=29, srv1481446541=18, srv715537649=87, srv2069095423=55, srv621387948=80, srv1107416503=5, srv1853554412=42, srv1830881713=38, srv1786151496=35, srv437163549=67, srv2031585610=52, srv1023108534=2, srv855388833=93, srv1505012112=19, srv53127904=75, srv2198996=58, srv743105488=88, srv1167273507=7, srv1463708527=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:49,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:49,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:49,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:49,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:49,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:49,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:49,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:49,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:49,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:49,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:49,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 07:54:49,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 07:54:49,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 07:54:49,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 07:54:49,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 07:54:49,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 07:54:49,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 07:54:49,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 07:54:49,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 07:54:49,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 07:54:49,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 07:54:49,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 07:54:49,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040176378808217825, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05589994267859546, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040176378808217825, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:49,536 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 07:54:49,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18290865196216763, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040176378808217825, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05589994267859546, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040176378808217825, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 07:54:49,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations. Found a solution that moves 2650 regions; Going from a computed imbalance of 0.18290865196216763 to a new imbalance of 0.015957817229309845. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020607435970032995); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03392075004994166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020607435970032995); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:49,800 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2127694272:77 , srv1380503240:80 , srv437163549:80 , srv1463708527:81 , srv855388833:81 , srv509414127:82 , srv70009736:82 , srv715537649:82 , srv743105488:82 , srv104349979:83 , srv1353198444:83 , srv1740068716:83 , srv1766145228:83 , srv1920773618:83 , srv2119918665:83 , srv319563501:83 , srv449196240:83 , srv548895116:83 , srv572617613:83 , srv911531193:83 , srv1109186877:84 , srv1579415360:84 , srv1621184483:84 , srv1686211774:84 , srv1773490758:84 , srv1824657958:84 , srv1937334152:84 , srv2047280228:84 , srv293337065:84 , srv38903024:84 , srv457603202:84 , srv1107416503:85 , srv1278814494:85 , srv1565709319:85 , srv158347759:85 , srv1703894567:85 , srv257484949:85 , srv283087146:85 , srv465043560:85 , srv627570515:85 , srv654865882:85 , srv714843958:85 , srv815835816:85 , srv817754264:85 , srv967630840:85 , srv1009037560:86 , srv1353494194:86 , srv1459778331:86 , srv1505012112:86 , srv160915758:86 , srv1615119545:86 , srv1842450759:86 , srv1874659565:86 , srv1883140587:86 , srv1884940367:86 , srv1909668126:86 , srv2031585610:86 , srv365235180:86 , srv450545153:86 , srv53127904:86 , srv672471253:86 , srv818564504:86 , srv1194503995:87 , srv1547679274:87 , srv1717989968:87 , srv1830881713:87 , srv1849633328:87 , srv2069095423:87 , srv507617960:87 , srv638931588:87 , srv84340461:87 , srv988571947:87 , srv1404385260:88 , srv1481446541:88 , srv1580011922:88 , srv158202948:88 , srv1786151496:88 , srv1853554412:88 , srv2055629832:88 , srv2198996:88 , srv324470022:88 , srv522784977:88 , srv874973751:88 , srv90803343:88 , srv1009017103:89 , srv1023108534:89 , srv1075423230:89 , srv1314998931:89 , srv1795908067:89 , srv1947474154:89 , srv284634369:89 , srv582167989:89 , srv1838793566:90 , srv608785619:90 , srv621387948:90 , srv1877694213:91 , srv2024553986:91 , srv999250868:91 , srv1167273507:93 , srv1276839472:510 } 2023-10-24 07:54:49,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:49,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1877694213=44, srv1909668126=47, srv1009017103=0, srv1314998931=11, srv1920773618=48, srv1937334152=49, srv672471253=84, srv714843958=86, srv818564504=91, srv911531193=96, srv1380503240=14, srv1874659565=43, srv1838793566=39, srv90803343=95, srv1615119545=27, srv582167989=78, srv1824657958=37, srv284634369=61, srv1795908067=36, srv1947474154=50, srv70009736=85, srv522784977=74, srv1459778331=16, srv450545153=69, srv509414127=73, srv1740068716=32, srv257484949=59, srv465043560=71, srv1353494194=13, srv638931588=82, srv1579415360=22, srv2055629832=54, srv967630840=97, srv1884940367=46, srv160915758=26, srv158202948=24, srv1278814494=10, srv548895116=76, srv84340461=92, srv1773490758=34, srv1353198444=12, srv158347759=25, srv1849633328=41, srv365235180=65, srv507617960=72, srv2024553986=51, srv815835816=89, srv449196240=68, srv1547679274=20, srv817754264=90, srv1580011922=23, srv2047280228=53, srv988571947=98, srv1276839472=9, srv104349979=3, srv1842450759=40, srv324470022=64, srv1883140587=45, srv608785619=79, srv293337065=62, srv874973751=94, srv1766145228=33, srv1703894567=30, srv38903024=66, srv1009037560=1, srv283087146=60, srv1565709319=21, srv654865882=83, srv1717989968=31, srv2127694272=57, srv2119918665=56, srv319563501=63, srv1109186877=6, srv999250868=99, srv572617613=77, srv627570515=81, srv457603202=70, srv1075423230=4, srv1621184483=28, srv1194503995=8, srv1404385260=15, srv1686211774=29, srv1481446541=18, srv715537649=87, srv2069095423=55, srv621387948=80, srv1107416503=5, srv1853554412=42, srv1830881713=38, srv1786151496=35, srv437163549=67, srv2031585610=52, srv1023108534=2, srv855388833=93, srv1505012112=19, srv53127904=75, srv2198996=58, srv743105488=88, srv1167273507=7, srv1463708527=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:49,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:49,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 07:54:49,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 07:54:49,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 07:54:49,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 07:54:49,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 07:54:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 07:54:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 07:54:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 07:54:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 07:54:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 07:54:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 07:54:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 07:54:50,007 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020607435970032995); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03392075004994166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020607435970032995); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:50,007 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 07:54:50,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01593926153271099, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020607435970032995); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03392075004994166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020607435970032995); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 07:54:50,290 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations. Found a solution that moves 998 regions; Going from a computed imbalance of 0.01593926153271099 to a new imbalance of 7.972595506571087E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006112386358117148); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00735503370238284); MoveCostFunction : (multiplier=7.0, imbalance=0.11088888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006112386358117148); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:50,292 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1580011922:81 , srv1766145228:82 , srv2127694272:82 , srv621387948:82 , srv999250868:82 , srv1167273507:83 , srv1877694213:83 , srv1920773618:83 , srv284634369:83 , srv293337065:83 , srv457603202:83 , srv572617613:83 , srv638931588:83 , srv1023108534:84 , srv1194503995:84 , srv1463708527:84 , srv158202948:84 , srv160915758:84 , srv1686211774:84 , srv1703894567:84 , srv1795908067:84 , srv1830881713:84 , srv1838793566:84 , srv2024553986:84 , srv324470022:84 , srv582167989:84 , srv608785619:84 , srv654865882:84 , srv817754264:84 , srv967630840:84 , srv1009017103:85 , srv1009037560:85 , srv1459778331:85 , srv1481446541:85 , srv1547679274:85 , srv158347759:85 , srv1824657958:85 , srv1842450759:85 , srv1849633328:85 , srv1853554412:85 , srv1874659565:85 , srv1884940367:85 , srv1909668126:85 , srv1947474154:85 , srv2119918665:85 , srv257484949:85 , srv283087146:85 , srv449196240:85 , srv522784977:85 , srv627570515:85 , srv815835816:85 , srv855388833:85 , srv874973751:85 , srv1107416503:86 , srv1109186877:86 , srv1314998931:86 , srv1353198444:86 , srv1380503240:86 , srv1404385260:86 , srv1505012112:86 , srv1565709319:86 , srv1621184483:86 , srv1883140587:86 , srv2055629832:86 , srv2198996:86 , srv319563501:86 , srv365235180:86 , srv38903024:86 , srv450545153:86 , srv507617960:86 , srv509414127:86 , srv672471253:86 , srv70009736:86 , srv715537649:86 , srv84340461:86 , srv988571947:86 , srv1075423230:87 , srv1278814494:87 , srv1353494194:87 , srv1579415360:87 , srv1717989968:87 , srv1740068716:87 , srv1786151496:87 , srv2031585610:87 , srv2047280228:87 , srv2069095423:87 , srv465043560:87 , srv53127904:87 , srv548895116:87 , srv90803343:87 , srv104349979:88 , srv1773490758:88 , srv1937334152:88 , srv437163549:88 , srv714843958:88 , srv743105488:88 , srv818564504:88 , srv911531193:88 , srv1615119545:89 , srv1276839472:555 } 2023-10-24 07:54:50,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:50,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1877694213=44, srv1909668126=47, srv1009017103=0, srv1314998931=11, srv1920773618=48, srv1937334152=49, srv672471253=84, srv714843958=86, srv818564504=91, srv911531193=96, srv1380503240=14, srv1874659565=43, srv1838793566=39, srv90803343=95, srv1615119545=27, srv582167989=78, srv1824657958=37, srv284634369=61, srv1795908067=36, srv1947474154=50, srv70009736=85, srv522784977=74, srv1459778331=16, srv450545153=69, srv509414127=73, srv1740068716=32, srv257484949=59, srv465043560=71, srv1353494194=13, srv638931588=82, srv1579415360=22, srv2055629832=54, srv967630840=97, srv1884940367=46, srv160915758=26, srv158202948=24, srv1278814494=10, srv548895116=76, srv84340461=92, srv1773490758=34, srv1353198444=12, srv158347759=25, srv1849633328=41, srv365235180=65, srv507617960=72, srv2024553986=51, srv815835816=89, srv449196240=68, srv1547679274=20, srv817754264=90, srv1580011922=23, srv2047280228=53, srv988571947=98, srv1276839472=9, srv104349979=3, srv1842450759=40, srv324470022=64, srv1883140587=45, srv608785619=79, srv293337065=62, srv874973751=94, srv1766145228=33, srv1703894567=30, srv38903024=66, srv1009037560=1, srv283087146=60, srv1565709319=21, srv654865882=83, srv1717989968=31, srv2127694272=57, srv2119918665=56, srv319563501=63, srv1109186877=6, srv999250868=99, srv572617613=77, srv627570515=81, srv457603202=70, srv1075423230=4, srv1621184483=28, srv1194503995=8, srv1404385260=15, srv1686211774=29, srv1481446541=18, srv715537649=87, srv2069095423=55, srv621387948=80, srv1107416503=5, srv1853554412=42, srv1830881713=38, srv1786151496=35, srv437163549=67, srv2031585610=52, srv1023108534=2, srv855388833=93, srv1505012112=19, srv53127904=75, srv2198996=58, srv743105488=88, srv1167273507=7, srv1463708527=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:50,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:50,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 07:54:50,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 07:54:50,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 07:54:50,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 07:54:50,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 07:54:50,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 07:54:50,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 07:54:50,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 07:54:50,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 07:54:50,453 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006112386358117148); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00735503370238284); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006112386358117148); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:50,453 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 07:54:50,454 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.902714052964828E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006112386358117148); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00735503370238284); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006112386358117148); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 07:54:50,820 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 488 ms to try 5000 different iterations. Found a solution that moves 277 regions; Going from a computed imbalance of 7.902714052964828E-4 to a new imbalance of 4.446445327240795E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.03077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:50,820 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv988571947:82 , srv1565709319:83 , srv1786151496:83 , srv1937334152:83 , srv714843958:83 , srv815835816:83 , srv104349979:84 , srv1278814494:84 , srv1314998931:84 , srv1380503240:84 , srv1481446541:84 , srv1579415360:84 , srv1580011922:84 , srv158202948:84 , srv158347759:84 , srv1615119545:84 , srv1686211774:84 , srv1717989968:84 , srv1740068716:84 , srv1773490758:84 , srv1795908067:84 , srv1853554412:84 , srv1874659565:84 , srv1883140587:84 , srv1884940367:84 , srv2024553986:84 , srv2047280228:84 , srv2069095423:84 , srv2119918665:84 , srv2127694272:84 , srv2198996:84 , srv283087146:84 , srv319563501:84 , srv365235180:84 , srv507617960:84 , srv522784977:84 , srv582167989:84 , srv621387948:84 , srv627570515:84 , srv715537649:84 , srv743105488:84 , srv818564504:84 , srv84340461:84 , srv855388833:84 , srv874973751:84 , srv1009017103:85 , srv1009037560:85 , srv1075423230:85 , srv1107416503:85 , srv1109186877:85 , srv1167273507:85 , srv1194503995:85 , srv1353198444:85 , srv1353494194:85 , srv1404385260:85 , srv1459778331:85 , srv1463708527:85 , srv1505012112:85 , srv1547679274:85 , srv160915758:85 , srv1621184483:85 , srv1766145228:85 , srv1830881713:85 , srv1838793566:85 , srv1849633328:85 , srv1877694213:85 , srv1909668126:85 , srv1920773618:85 , srv1947474154:85 , srv2055629832:85 , srv257484949:85 , srv284634369:85 , srv293337065:85 , srv324470022:85 , srv38903024:85 , srv437163549:85 , srv449196240:85 , srv450545153:85 , srv457603202:85 , srv465043560:85 , srv509414127:85 , srv53127904:85 , srv548895116:85 , srv572617613:85 , srv608785619:85 , srv638931588:85 , srv654865882:85 , srv672471253:85 , srv70009736:85 , srv817754264:85 , srv90803343:85 , srv967630840:85 , srv1023108534:86 , srv1703894567:86 , srv1824657958:86 , srv1842450759:86 , srv2031585610:86 , srv911531193:86 , srv999250868:86 , srv1276839472:630 } 2023-10-24 07:54:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1877694213=44, srv1909668126=47, srv1009017103=0, srv1314998931=11, srv1920773618=48, srv1937334152=49, srv672471253=84, srv714843958=86, srv818564504=91, srv911531193=96, srv1380503240=14, srv1874659565=43, srv1838793566=39, srv90803343=95, srv1615119545=27, srv582167989=78, srv1824657958=37, srv284634369=61, srv1795908067=36, srv1947474154=50, srv70009736=85, srv522784977=74, srv1459778331=16, srv450545153=69, srv509414127=73, srv1740068716=32, srv257484949=59, srv465043560=71, srv1353494194=13, srv638931588=82, srv1579415360=22, srv2055629832=54, srv967630840=97, srv1884940367=46, srv160915758=26, srv158202948=24, srv1278814494=10, srv548895116=76, srv84340461=92, srv1773490758=34, srv1353198444=12, srv158347759=25, srv1849633328=41, srv365235180=65, srv507617960=72, srv2024553986=51, srv815835816=89, srv449196240=68, srv1547679274=20, srv817754264=90, srv1580011922=23, srv2047280228=53, srv988571947=98, srv1276839472=9, srv104349979=3, srv1842450759=40, srv324470022=64, srv1883140587=45, srv608785619=79, srv293337065=62, srv874973751=94, srv1766145228=33, srv1703894567=30, srv38903024=66, srv1009037560=1, srv283087146=60, srv1565709319=21, srv654865882=83, srv1717989968=31, srv2127694272=57, srv2119918665=56, srv319563501=63, srv1109186877=6, srv999250868=99, srv572617613=77, srv627570515=81, srv457603202=70, srv1075423230=4, srv1621184483=28, srv1194503995=8, srv1404385260=15, srv1686211774=29, srv1481446541=18, srv715537649=87, srv2069095423=55, srv621387948=80, srv1107416503=5, srv1853554412=42, srv1830881713=38, srv1786151496=35, srv437163549=67, srv2031585610=52, srv1023108534=2, srv855388833=93, srv1505012112=19, srv53127904=75, srv2198996=58, srv743105488=88, srv1167273507=7, srv1463708527=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:50,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:50,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 07:54:50,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 07:54:50,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 07:54:50,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 07:54:50,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 07:54:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 07:54:50,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 07:54:50,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 07:54:50,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 07:54:50,951 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:50,951 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-24 07:54:50,951 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.252485781660302E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-24 07:54:51,229 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 379 ms to try 5000 different iterations. Found a solution that moves 44 regions; Going from a computed imbalance of 4.252485781660302E-5 to a new imbalance of 7.3813338813670545E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.004888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:51,230 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv465043560:82 , srv1009017103:83 , srv1023108534:83 , srv1615119545:83 , srv1795908067:83 , srv1842450759:83 , srv1874659565:83 , srv2031585610:83 , srv257484949:83 , srv911531193:83 , srv1009037560:84 , srv104349979:84 , srv1075423230:84 , srv1107416503:84 , srv1109186877:84 , srv1167273507:84 , srv1194503995:84 , srv1278814494:84 , srv1314998931:84 , srv1353198444:84 , srv1353494194:84 , srv1380503240:84 , srv1404385260:84 , srv1459778331:84 , srv1481446541:84 , srv1505012112:84 , srv1547679274:84 , srv1579415360:84 , srv1580011922:84 , srv158202948:84 , srv158347759:84 , srv160915758:84 , srv1621184483:84 , srv1703894567:84 , srv1717989968:84 , srv1740068716:84 , srv1766145228:84 , srv1773490758:84 , srv1786151496:84 , srv1824657958:84 , srv1830881713:84 , srv1838793566:84 , srv1849633328:84 , srv1853554412:84 , srv1877694213:84 , srv1884940367:84 , srv1909668126:84 , srv1920773618:84 , srv1947474154:84 , srv2024553986:84 , srv2055629832:84 , srv2069095423:84 , srv2119918665:84 , srv2198996:84 , srv283087146:84 , srv284634369:84 , srv293337065:84 , srv319563501:84 , srv324470022:84 , srv365235180:84 , srv38903024:84 , srv437163549:84 , srv449196240:84 , srv450545153:84 , srv457603202:84 , srv507617960:84 , srv509414127:84 , srv522784977:84 , srv53127904:84 , srv548895116:84 , srv572617613:84 , srv582167989:84 , srv608785619:84 , srv638931588:84 , srv654865882:84 , srv672471253:84 , srv70009736:84 , srv714843958:84 , srv715537649:84 , srv743105488:84 , srv817754264:84 , srv818564504:84 , srv84340461:84 , srv855388833:84 , srv874973751:84 , srv90803343:84 , srv967630840:84 , srv988571947:84 , srv999250868:84 , srv1463708527:85 , srv1565709319:85 , srv1686211774:85 , srv1883140587:85 , srv1937334152:85 , srv2047280228:85 , srv2127694272:85 , srv621387948:85 , srv627570515:85 , srv815835816:86 , srv1276839472:684 } 2023-10-24 07:54:51,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:51,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1877694213=44, srv1909668126=47, srv1009017103=0, srv1314998931=11, srv1920773618=48, srv1937334152=49, srv672471253=84, srv714843958=86, srv818564504=91, srv911531193=96, srv1380503240=14, srv1874659565=43, srv1838793566=39, srv90803343=95, srv1615119545=27, srv582167989=78, srv1824657958=37, srv284634369=61, srv1795908067=36, srv1947474154=50, srv70009736=85, srv522784977=74, srv1459778331=16, srv450545153=69, srv509414127=73, srv1740068716=32, srv257484949=59, srv465043560=71, srv1353494194=13, srv638931588=82, srv1579415360=22, srv2055629832=54, srv967630840=97, srv1884940367=46, srv160915758=26, srv158202948=24, srv1278814494=10, srv548895116=76, srv84340461=92, srv1773490758=34, srv1353198444=12, srv158347759=25, srv1849633328=41, srv365235180=65, srv507617960=72, srv2024553986=51, srv815835816=89, srv449196240=68, srv1547679274=20, srv817754264=90, srv1580011922=23, srv2047280228=53, srv988571947=98, srv1276839472=9, srv104349979=3, srv1842450759=40, srv324470022=64, srv1883140587=45, srv608785619=79, srv293337065=62, srv874973751=94, srv1766145228=33, srv1703894567=30, srv38903024=66, srv1009037560=1, srv283087146=60, srv1565709319=21, srv654865882=83, srv1717989968=31, srv2127694272=57, srv2119918665=56, srv319563501=63, srv1109186877=6, srv999250868=99, srv572617613=77, srv627570515=81, srv457603202=70, srv1075423230=4, srv1621184483=28, srv1194503995=8, srv1404385260=15, srv1686211774=29, srv1481446541=18, srv715537649=87, srv2069095423=55, srv621387948=80, srv1107416503=5, srv1853554412=42, srv1830881713=38, srv1786151496=35, srv437163549=67, srv2031585610=52, srv1023108534=2, srv855388833=93, srv1505012112=19, srv53127904=75, srv2198996=58, srv743105488=88, srv1167273507=7, srv1463708527=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:51,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:51,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-24 07:54:51,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-24 07:54:51,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-24 07:54:51,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-24 07:54:51,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-24 07:54:51,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-24 07:54:51,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-24 07:54:51,411 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:51,412 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv465043560:82 , srv1009017103:83 , srv1023108534:83 , srv1615119545:83 , srv1795908067:83 , srv1842450759:83 , srv1874659565:83 , srv2031585610:83 , srv257484949:83 , srv911531193:83 , srv1009037560:84 , srv104349979:84 , srv1075423230:84 , srv1107416503:84 , srv1109186877:84 , srv1167273507:84 , srv1194503995:84 , srv1278814494:84 , srv1314998931:84 , srv1353198444:84 , srv1353494194:84 , srv1380503240:84 , srv1404385260:84 , srv1459778331:84 , srv1481446541:84 , srv1505012112:84 , srv1547679274:84 , srv1579415360:84 , srv1580011922:84 , srv158202948:84 , srv158347759:84 , srv160915758:84 , srv1621184483:84 , srv1703894567:84 , srv1717989968:84 , srv1740068716:84 , srv1766145228:84 , srv1773490758:84 , srv1786151496:84 , srv1824657958:84 , srv1830881713:84 , srv1838793566:84 , srv1849633328:84 , srv1853554412:84 , srv1877694213:84 , srv1884940367:84 , srv1909668126:84 , srv1920773618:84 , srv1947474154:84 , srv2024553986:84 , srv2055629832:84 , srv2069095423:84 , srv2119918665:84 , srv2198996:84 , srv283087146:84 , srv284634369:84 , srv293337065:84 , srv319563501:84 , srv324470022:84 , srv365235180:84 , srv38903024:84 , srv437163549:84 , srv449196240:84 , srv450545153:84 , srv457603202:84 , srv507617960:84 , srv509414127:84 , srv522784977:84 , srv53127904:84 , srv548895116:84 , srv572617613:84 , srv582167989:84 , srv608785619:84 , srv638931588:84 , srv654865882:84 , srv672471253:84 , srv70009736:84 , srv714843958:84 , srv715537649:84 , srv743105488:84 , srv817754264:84 , srv818564504:84 , srv84340461:84 , srv855388833:84 , srv874973751:84 , srv90803343:84 , srv967630840:84 , srv988571947:84 , srv999250868:84 , srv1463708527:85 , srv1565709319:85 , srv1686211774:85 , srv1883140587:85 , srv1937334152:85 , srv2047280228:85 , srv2127694272:85 , srv621387948:85 , srv627570515:85 , srv815835816:86 , srv1276839472:684 } 2023-10-24 07:54:51,436 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 304) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10651 (was 11178) 2023-10-24 07:54:51,443 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=170, AvailableMemoryMB=10650 2023-10-24 07:54:51,444 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-24 07:54:51,444 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-24 07:54:51,452 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1040929842:84 , srv1123772767:84 , srv1153831148:84 , srv1182982952:84 , srv1187149772:84 , srv1265879205:84 , srv1295809052:84 , srv1405541776:84 , srv1536318583:84 , srv1567025008:84 , srv1626889807:84 , srv1637230184:84 , srv1648623142:84 , srv1730438441:84 , srv1761498363:84 , srv1951508502:84 , srv1964817437:84 , srv1967082257:84 , srv1995888145:84 , srv2066944093:84 , srv273547469:84 , srv303906884:84 , srv351287418:84 , srv474157465:84 , srv602061771:84 , srv786742255:84 , srv799149444:84 , srv902563415:84 , srv993714832:84 , srv169966030:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-24 07:54:51,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:54:51,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1648623142=12, srv1040929842=0, srv786742255=26, srv1295809052=6, srv1995888145=19, srv303906884=22, srv1536318583=8, srv351287418=23, srv1567025008=9, srv799149444=27, srv1637230184=11, srv474157465=24, srv1951508502=16, srv1964817437=17, srv993714832=29, srv1265879205=5, srv1405541776=7, srv1182982952=3, srv169966030=13, srv1730438441=14, srv1123772767=1, srv1187149772=4, srv902563415=28, srv1626889807=10, srv1153831148=2, srv602061771=25, srv1967082257=18, srv273547469=21, srv1761498363=15, srv2066944093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:54:51,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:54:51,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-24 07:54:51,480 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.122980410139973, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:54:51,480 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912580550520952, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.122980410139973, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-24 07:56:51,467 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5630912 different iterations. Found a solution that moves 1867 regions; Going from a computed imbalance of 0.9912580550520952 to a new imbalance of 4.3576711383728133E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6914814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:56:51,488 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1040929842:90 , srv1123772767:90 , srv1153831148:90 , srv1182982952:90 , srv1187149772:90 , srv1265879205:90 , srv1295809052:90 , srv1405541776:90 , srv1536318583:90 , srv1567025008:90 , srv1626889807:90 , srv1637230184:90 , srv1648623142:90 , srv169966030:90 , srv1730438441:90 , srv1761498363:90 , srv1951508502:90 , srv1964817437:90 , srv1967082257:90 , srv1995888145:90 , srv2066944093:90 , srv273547469:90 , srv303906884:90 , srv351287418:90 , srv474157465:90 , srv602061771:90 , srv786742255:90 , srv799149444:90 , srv902563415:90 , srv993714832:90 } 2023-10-24 07:56:51,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-24 07:56:51,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1648623142=12, srv1040929842=0, srv786742255=26, srv1295809052=6, srv1995888145=19, srv303906884=22, srv1536318583=8, srv351287418=23, srv1567025008=9, srv799149444=27, srv1637230184=11, srv474157465=24, srv1951508502=16, srv1964817437=17, srv993714832=29, srv1265879205=5, srv1405541776=7, srv1182982952=3, srv169966030=13, srv1730438441=14, srv1123772767=1, srv1187149772=4, srv902563415=28, srv1626889807=10, srv1153831148=2, srv602061771=25, srv1967082257=18, srv273547469=21, srv1761498363=15, srv2066944093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-24 07:56:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-24 07:56:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-24 07:56:51,514 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-24 07:56:51,515 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1040929842:90 , srv1123772767:90 , srv1153831148:90 , srv1182982952:90 , srv1187149772:90 , srv1265879205:90 , srv1295809052:90 , srv1405541776:90 , srv1536318583:90 , srv1567025008:90 , srv1626889807:90 , srv1637230184:90 , srv1648623142:90 , srv169966030:90 , srv1730438441:90 , srv1761498363:90 , srv1951508502:90 , srv1964817437:90 , srv1967082257:90 , srv1995888145:90 , srv2066944093:90 , srv273547469:90 , srv303906884:90 , srv351287418:90 , srv474157465:90 , srv602061771:90 , srv786742255:90 , srv799149444:90 , srv902563415:90 , srv993714832:90 } 2023-10-24 07:56:51,536 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 344), ProcessCount=167 (was 170), AvailableMemoryMB=10533 (was 10650)