2023-07-28 23:57:00,030 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-28 23:57:00,437 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-28 23:57:00,593 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-28 23:57:00,681 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-28 23:57:00,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-28 23:57:00,730 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=630, ProcessCount=185, AvailableMemoryMB=3189 2023-07-28 23:57:00,734 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-28 23:57:00,735 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-28 23:57:00,885 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010830925:84 , srv1066876137:84 , srv1086151291:84 , srv1086677687:84 , srv1140002189:84 , srv1158681975:84 , srv1201326999:84 , srv1207134992:84 , srv1221806104:84 , srv1236121429:84 , srv1268852304:84 , srv1289761801:84 , srv1294081917:84 , srv1306867584:84 , srv1323962104:84 , srv1329889172:84 , srv1343745011:84 , srv1373771051:84 , srv1402833823:84 , srv1418457989:84 , srv1423619596:84 , srv1442906609:84 , srv1451464229:84 , srv1485015063:84 , srv1488533369:84 , srv1534049159:84 , srv154687073:84 , srv1557139706:84 , srv1597332442:84 , srv1637868355:84 , srv164032000:84 , srv1642556172:84 , srv1645288349:84 , srv1663087791:84 , srv1668566051:84 , srv1759136045:84 , srv1785690214:84 , srv1789765528:84 , srv1827046290:84 , srv1844017590:84 , srv1846964592:84 , srv1853938252:84 , srv1943972075:84 , srv1945724048:84 , srv2010705807:84 , srv2017720347:84 , srv2025817003:84 , srv2028817560:84 , srv2069338684:84 , srv2095337228:84 , srv215785232:84 , srv261849072:84 , srv267514320:84 , srv273719227:84 , srv276550531:84 , srv291222628:84 , srv29746775:84 , srv319801855:84 , srv328507217:84 , srv333959821:84 , srv334989951:84 , srv337005277:84 , srv353771232:84 , srv353876294:84 , srv394035782:84 , srv398783269:84 , srv420682597:84 , srv441656790:84 , srv442754546:84 , srv447210823:84 , srv453417817:84 , srv464030381:84 , srv464337054:84 , srv479478026:84 , srv532474436:84 , srv546375539:84 , srv549365310:84 , srv596931509:84 , srv612416283:84 , srv617160874:84 , srv631378781:84 , srv637325040:84 , srv640441625:84 , srv668585479:84 , srv677109922:84 , srv679371237:84 , srv708959949:84 , srv729047074:84 , srv740316064:84 , srv790884195:84 , srv806016165:84 , srv82590628:84 , srv834132485:84 , srv885983339:84 , srv911737394:84 , srv914385452:84 , srv972640317:84 , srv977737667:84 , srv994033504:84 , srv1257620982:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-28 23:57:00,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:00,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2025817003=47, srv1827046290=39, srv1663087791=34, srv333959821=60, srv994033504=99, srv398783269=66, srv806016165=91, srv885983339=94, srv1257620982=10, srv2095337228=50, srv740316064=89, srv1268852304=11, srv1418457989=20, srv668585479=84, srv273719227=54, srv334989951=61, srv261849072=52, srv164032000=31, srv834132485=93, srv276550531=55, srv82590628=92, srv1402833823=19, srv1010830925=0, srv1294081917=13, srv442754546=69, srv1140002189=4, srv596931509=78, srv1086151291=2, srv1485015063=24, srv1306867584=14, srv1844017590=40, srv1637868355=30, srv394035782=65, srv1158681975=5, srv1759136045=36, srv1488533369=25, srv479478026=74, srv1373771051=18, srv1846964592=41, srv1329889172=16, srv1289761801=12, srv1201326999=6, srv617160874=80, srv1221806104=8, srv1236121429=9, srv1451464229=23, srv464337054=73, srv464030381=72, srv1086677687=3, srv1423619596=21, srv790884195=90, srv1557139706=28, srv1789765528=38, srv532474436=75, srv319801855=58, srv1343745011=17, srv2028817560=48, srv2069338684=49, srv546375539=76, srv453417817=71, srv637325040=82, srv328507217=59, srv1207134992=7, srv631378781=81, srv679371237=86, srv337005277=62, srv1534049159=26, srv914385452=96, srv1668566051=35, srv29746775=57, srv1945724048=44, srv612416283=79, srv1442906609=22, srv1597332442=29, srv267514320=53, srv911737394=95, srv1853938252=42, srv549365310=77, srv1323962104=15, srv640441625=83, srv291222628=56, srv972640317=97, srv353771232=63, srv1642556172=32, srv353876294=64, srv447210823=70, srv677109922=85, srv420682597=67, srv977737667=98, srv1645288349=33, srv2017720347=46, srv441656790=68, srv729047074=88, srv1066876137=1, srv1785690214=37, srv1943972075=43, srv215785232=51, srv708959949=87, srv2010705807=45, srv154687073=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:01,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:01,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:01,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:01,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 23:57:01,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 23:57:01,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 23:57:01,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 23:57:01,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 23:57:01,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 23:57:01,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 23:57:01,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 23:57:01,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 23:57:01,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 23:57:01,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 23:57:01,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 23:57:01,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 23:57:01,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 23:57:01,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 23:57:01,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 23:57:01,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 23:57:01,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 23:57:01,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 23:57:01,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 23:57:01,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 23:57:01,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 23:57:01,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 23:57:01,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 23:57:01,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 23:57:01,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 23:57:01,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 23:57:01,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 23:57:01,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 23:57:01,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 23:57:01,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 23:57:01,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 23:57:01,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 23:57:01,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 23:57:01,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 23:57:01,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 23:57:01,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 23:57:01,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 23:57:01,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 23:57:01,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 23:57:01,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 23:57:01,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 23:57:01,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 23:57:01,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 23:57:01,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 23:57:01,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 23:57:01,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 23:57:01,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 23:57:01,441 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:01,441 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 23:57:01,442 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 23:57:01,878 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 812 ms to try 5000 different iterations. Found a solution that moves 4691 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17756922669284317. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038788551013628686, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055579792692848566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5212222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038788551013628686, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:01,895 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1373771051:83 , srv1759136045:83 , srv1207134992:84 , srv1268852304:84 , srv1423619596:84 , srv164032000:84 , srv2069338684:84 , srv394035782:84 , srv464337054:84 , srv631378781:84 , srv790884195:84 , srv885983339:84 , srv1066876137:85 , srv1158681975:85 , srv1201326999:85 , srv1418457989:85 , srv1789765528:85 , srv1844017590:85 , srv1943972075:85 , srv1945724048:85 , srv267514320:85 , srv276550531:85 , srv333959821:85 , srv398783269:85 , srv420682597:85 , srv453417817:85 , srv464030381:85 , srv637325040:85 , srv640441625:85 , srv708959949:85 , srv834132485:85 , srv911737394:85 , srv914385452:85 , srv977737667:85 , srv1221806104:86 , srv1236121429:86 , srv1289761801:86 , srv1294081917:86 , srv1329889172:86 , srv1485015063:86 , srv1642556172:86 , srv1668566051:86 , srv1846964592:86 , srv2010705807:86 , srv2017720347:86 , srv2025817003:86 , srv291222628:86 , srv29746775:86 , srv334989951:86 , srv337005277:86 , srv353771232:86 , srv442754546:86 , srv479478026:86 , srv532474436:86 , srv596931509:86 , srv668585479:86 , srv677109922:86 , srv740316064:86 , srv82590628:86 , srv972640317:86 , srv1086677687:87 , srv1140002189:87 , srv1306867584:87 , srv1323962104:87 , srv1343745011:87 , srv1534049159:87 , srv1645288349:87 , srv1663087791:87 , srv353876294:87 , srv546375539:87 , srv612416283:87 , srv617160874:87 , srv806016165:87 , srv1086151291:88 , srv1402833823:88 , srv1451464229:88 , srv154687073:88 , srv1557139706:88 , srv1597332442:88 , srv1785690214:88 , srv1827046290:88 , srv2028817560:88 , srv215785232:88 , srv261849072:88 , srv441656790:88 , srv549365310:88 , srv328507217:89 , srv447210823:89 , srv994033504:89 , srv1010830925:90 , srv1637868355:90 , srv1853938252:90 , srv273719227:90 , srv679371237:90 , srv319801855:91 , srv1442906609:92 , srv1488533369:92 , srv729047074:92 , srv2095337228:94 , srv1257620982:435 } 2023-07-28 23:57:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2025817003=47, srv1827046290=39, srv1663087791=34, srv333959821=60, srv994033504=99, srv398783269=66, srv806016165=91, srv885983339=94, srv1257620982=10, srv2095337228=50, srv740316064=89, srv1268852304=11, srv1418457989=20, srv668585479=84, srv273719227=54, srv334989951=61, srv261849072=52, srv164032000=31, srv834132485=93, srv276550531=55, srv82590628=92, srv1402833823=19, srv1010830925=0, srv1294081917=13, srv442754546=69, srv1140002189=4, srv596931509=78, srv1086151291=2, srv1485015063=24, srv1306867584=14, srv1844017590=40, srv1637868355=30, srv394035782=65, srv1158681975=5, srv1759136045=36, srv1488533369=25, srv479478026=74, srv1373771051=18, srv1846964592=41, srv1329889172=16, srv1289761801=12, srv1201326999=6, srv617160874=80, srv1221806104=8, srv1236121429=9, srv1451464229=23, srv464337054=73, srv464030381=72, srv1086677687=3, srv1423619596=21, srv790884195=90, srv1557139706=28, srv1789765528=38, srv532474436=75, srv319801855=58, srv1343745011=17, srv2028817560=48, srv2069338684=49, srv546375539=76, srv453417817=71, srv637325040=82, srv328507217=59, srv1207134992=7, srv631378781=81, srv679371237=86, srv337005277=62, srv1534049159=26, srv914385452=96, srv1668566051=35, srv29746775=57, srv1945724048=44, srv612416283=79, srv1442906609=22, srv1597332442=29, srv267514320=53, srv911737394=95, srv1853938252=42, srv549365310=77, srv1323962104=15, srv640441625=83, srv291222628=56, srv972640317=97, srv353771232=63, srv1642556172=32, srv353876294=64, srv447210823=70, srv677109922=85, srv420682597=67, srv977737667=98, srv1645288349=33, srv2017720347=46, srv441656790=68, srv729047074=88, srv1066876137=1, srv1785690214=37, srv1943972075=43, srv215785232=51, srv708959949=87, srv2010705807=45, srv154687073=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:01,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:01,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:01,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:01,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:01,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:01,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:01,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:01,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:01,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:01,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:01,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:01,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:01,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:01,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:01,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:01,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:01,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:01,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 23:57:01,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 23:57:01,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 23:57:01,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 23:57:01,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 23:57:01,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 23:57:01,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 23:57:01,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 23:57:01,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 23:57:01,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 23:57:01,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 23:57:01,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 23:57:01,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 23:57:01,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 23:57:01,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 23:57:01,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 23:57:01,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 23:57:01,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 23:57:02,142 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038788551013628686, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055579792692848566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038788551013628686, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:02,142 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 23:57:02,143 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17753637960878835, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038788551013628686, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055579792692848566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038788551013628686, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 23:57:02,402 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 383 ms to try 5000 different iterations. Found a solution that moves 2660 regions; Going from a computed imbalance of 0.17753637960878835 to a new imbalance of 0.01495425169771634. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020489702504350754); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020489702504350754); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:02,412 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2095337228:79 , srv319801855:79 , srv729047074:79 , srv1442906609:80 , srv1488533369:81 , srv1853938252:81 , srv679371237:81 , srv1010830925:82 , srv1402833823:82 , srv215785232:82 , srv1086151291:83 , srv1534049159:83 , srv1663087791:83 , srv1785690214:83 , srv2028817560:83 , srv328507217:83 , srv353876294:83 , srv479478026:83 , srv1086677687:84 , srv1597332442:84 , srv1637868355:84 , srv1827046290:84 , srv261849072:84 , srv612416283:84 , srv617160874:84 , srv677109922:84 , srv806016165:84 , srv994033504:84 , srv1221806104:85 , srv1294081917:85 , srv1306867584:85 , srv1323962104:85 , srv1329889172:85 , srv1343745011:85 , srv1451464229:85 , srv1789765528:85 , srv1945724048:85 , srv2017720347:85 , srv273719227:85 , srv334989951:85 , srv353771232:85 , srv546375539:85 , srv549365310:85 , srv596931509:85 , srv1066876137:86 , srv1140002189:86 , srv1289761801:86 , srv154687073:86 , srv1557139706:86 , srv1844017590:86 , srv1846964592:86 , srv1943972075:86 , srv276550531:86 , srv333959821:86 , srv420682597:86 , srv532474436:86 , srv640441625:86 , srv708959949:86 , srv911737394:86 , srv977737667:86 , srv1201326999:87 , srv1268852304:87 , srv1642556172:87 , srv1645288349:87 , srv1668566051:87 , srv291222628:87 , srv337005277:87 , srv441656790:87 , srv442754546:87 , srv447210823:87 , srv453417817:87 , srv464337054:87 , srv668585479:87 , srv82590628:87 , srv1158681975:88 , srv1207134992:88 , srv1236121429:88 , srv1418457989:88 , srv1485015063:88 , srv2010705807:88 , srv2025817003:88 , srv2069338684:88 , srv267514320:88 , srv29746775:88 , srv394035782:88 , srv398783269:88 , srv637325040:88 , srv740316064:88 , srv834132485:88 , srv1373771051:89 , srv1759136045:89 , srv790884195:89 , srv885983339:89 , srv914385452:89 , srv972640317:89 , srv1423619596:90 , srv164032000:90 , srv631378781:90 , srv464030381:91 , srv1257620982:521 } 2023-07-28 23:57:02,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:02,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2025817003=47, srv1827046290=39, srv1663087791=34, srv333959821=60, srv994033504=99, srv398783269=66, srv806016165=91, srv885983339=94, srv1257620982=10, srv2095337228=50, srv740316064=89, srv1268852304=11, srv1418457989=20, srv668585479=84, srv273719227=54, srv334989951=61, srv261849072=52, srv164032000=31, srv834132485=93, srv276550531=55, srv82590628=92, srv1402833823=19, srv1010830925=0, srv1294081917=13, srv442754546=69, srv1140002189=4, srv596931509=78, srv1086151291=2, srv1485015063=24, srv1306867584=14, srv1844017590=40, srv1637868355=30, srv394035782=65, srv1158681975=5, srv1759136045=36, srv1488533369=25, srv479478026=74, srv1373771051=18, srv1846964592=41, srv1329889172=16, srv1289761801=12, srv1201326999=6, srv617160874=80, srv1221806104=8, srv1236121429=9, srv1451464229=23, srv464337054=73, srv464030381=72, srv1086677687=3, srv1423619596=21, srv790884195=90, srv1557139706=28, srv1789765528=38, srv532474436=75, srv319801855=58, srv1343745011=17, srv2028817560=48, srv2069338684=49, srv546375539=76, srv453417817=71, srv637325040=82, srv328507217=59, srv1207134992=7, srv631378781=81, srv679371237=86, srv337005277=62, srv1534049159=26, srv914385452=96, srv1668566051=35, srv29746775=57, srv1945724048=44, srv612416283=79, srv1442906609=22, srv1597332442=29, srv267514320=53, srv911737394=95, srv1853938252=42, srv549365310=77, srv1323962104=15, srv640441625=83, srv291222628=56, srv972640317=97, srv353771232=63, srv1642556172=32, srv353876294=64, srv447210823=70, srv677109922=85, srv420682597=67, srv977737667=98, srv1645288349=33, srv2017720347=46, srv441656790=68, srv729047074=88, srv1066876137=1, srv1785690214=37, srv1943972075=43, srv215785232=51, srv708959949=87, srv2010705807=45, srv154687073=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:02,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:02,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:02,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:02,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:02,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:02,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:02,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:02,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:02,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:02,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:02,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:02,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:02,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 23:57:02,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 23:57:02,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 23:57:02,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 23:57:02,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 23:57:02,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 23:57:02,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 23:57:02,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 23:57:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 23:57:02,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 23:57:02,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 23:57:02,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 23:57:02,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 23:57:02,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 23:57:02,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020489702504350754); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020489702504350754); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:02,577 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 23:57:02,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014935625979620884, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020489702504350754); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020489702504350754); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 23:57:02,831 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 366 ms to try 5000 different iterations. Found a solution that moves 1027 regions; Going from a computed imbalance of 0.014935625979620884 to a new imbalance of 5.224461129179082E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007336359610916769); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00994936676326182); MoveCostFunction : (multiplier=7.0, imbalance=0.11411111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007336359610916769); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:02,832 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv447210823:81 , srv441656790:82 , srv972640317:82 , srv1158681975:83 , srv1236121429:83 , srv1485015063:83 , srv1645288349:83 , srv2010705807:83 , srv2025817003:83 , srv2095337228:83 , srv273719227:83 , srv464030381:83 , srv631378781:83 , srv1140002189:84 , srv1418457989:84 , srv1423619596:84 , srv1488533369:84 , srv154687073:84 , srv1637868355:84 , srv1642556172:84 , srv1827046290:84 , srv291222628:84 , srv29746775:84 , srv398783269:84 , srv532474436:84 , srv637325040:84 , srv740316064:84 , srv790884195:84 , srv82590628:84 , srv885983339:84 , srv914385452:84 , srv1201326999:85 , srv1207134992:85 , srv1221806104:85 , srv1289761801:85 , srv1306867584:85 , srv1323962104:85 , srv1343745011:85 , srv1373771051:85 , srv1442906609:85 , srv1451464229:85 , srv1557139706:85 , srv1597332442:85 , srv164032000:85 , srv1759136045:85 , srv2017720347:85 , srv2069338684:85 , srv261849072:85 , srv267514320:85 , srv333959821:85 , srv337005277:85 , srv442754546:85 , srv549365310:85 , srv668585479:85 , srv834132485:85 , srv994033504:85 , srv1010830925:86 , srv1066876137:86 , srv1086151291:86 , srv1329889172:86 , srv1668566051:86 , srv1785690214:86 , srv1844017590:86 , srv1846964592:86 , srv1853938252:86 , srv1943972075:86 , srv276550531:86 , srv328507217:86 , srv334989951:86 , srv353771232:86 , srv394035782:86 , srv453417817:86 , srv546375539:86 , srv596931509:86 , srv617160874:86 , srv677109922:86 , srv679371237:86 , srv708959949:86 , srv729047074:86 , srv806016165:86 , srv977737667:86 , srv1086677687:87 , srv1268852304:87 , srv1294081917:87 , srv1402833823:87 , srv1663087791:87 , srv1789765528:87 , srv1945724048:87 , srv2028817560:87 , srv353876294:87 , srv420682597:87 , srv464337054:87 , srv612416283:87 , srv640441625:87 , srv911737394:87 , srv1534049159:88 , srv319801855:88 , srv479478026:88 , srv215785232:89 , srv1257620982:567 } 2023-07-28 23:57:02,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:02,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2025817003=47, srv1827046290=39, srv1663087791=34, srv333959821=60, srv994033504=99, srv398783269=66, srv806016165=91, srv885983339=94, srv1257620982=10, srv2095337228=50, srv740316064=89, srv1268852304=11, srv1418457989=20, srv668585479=84, srv273719227=54, srv334989951=61, srv261849072=52, srv164032000=31, srv834132485=93, srv276550531=55, srv82590628=92, srv1402833823=19, srv1010830925=0, srv1294081917=13, srv442754546=69, srv1140002189=4, srv596931509=78, srv1086151291=2, srv1485015063=24, srv1306867584=14, srv1844017590=40, srv1637868355=30, srv394035782=65, srv1158681975=5, srv1759136045=36, srv1488533369=25, srv479478026=74, srv1373771051=18, srv1846964592=41, srv1329889172=16, srv1289761801=12, srv1201326999=6, srv617160874=80, srv1221806104=8, srv1236121429=9, srv1451464229=23, srv464337054=73, srv464030381=72, srv1086677687=3, srv1423619596=21, srv790884195=90, srv1557139706=28, srv1789765528=38, srv532474436=75, srv319801855=58, srv1343745011=17, srv2028817560=48, srv2069338684=49, srv546375539=76, srv453417817=71, srv637325040=82, srv328507217=59, srv1207134992=7, srv631378781=81, srv679371237=86, srv337005277=62, srv1534049159=26, srv914385452=96, srv1668566051=35, srv29746775=57, srv1945724048=44, srv612416283=79, srv1442906609=22, srv1597332442=29, srv267514320=53, srv911737394=95, srv1853938252=42, srv549365310=77, srv1323962104=15, srv640441625=83, srv291222628=56, srv972640317=97, srv353771232=63, srv1642556172=32, srv353876294=64, srv447210823=70, srv677109922=85, srv420682597=67, srv977737667=98, srv1645288349=33, srv2017720347=46, srv441656790=68, srv729047074=88, srv1066876137=1, srv1785690214=37, srv1943972075=43, srv215785232=51, srv708959949=87, srv2010705807=45, srv154687073=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:02,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:02,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:02,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:02,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:02,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 23:57:02,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 23:57:02,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 23:57:02,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 23:57:02,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 23:57:02,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 23:57:02,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 23:57:02,962 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007336359610916769); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00994936676326182); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007336359610916769); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:02,963 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 23:57:02,963 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.152549052171441E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007336359610916769); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00994936676326182); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007336359610916769); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 23:57:03,250 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 381 ms to try 5000 different iterations. Found a solution that moves 282 regions; Going from a computed imbalance of 5.152549052171441E-4 to a new imbalance of 2.8615500086436447E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.03133333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:03,252 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv154687073:83 , srv1557139706:83 , srv164032000:83 , srv1668566051:83 , srv215785232:83 , srv464030381:83 , srv1010830925:84 , srv1086151291:84 , srv1268852304:84 , srv1294081917:84 , srv1306867584:84 , srv1423619596:84 , srv1451464229:84 , srv1534049159:84 , srv1637868355:84 , srv1663087791:84 , srv2028817560:84 , srv267514320:84 , srv273719227:84 , srv29746775:84 , srv319801855:84 , srv328507217:84 , srv337005277:84 , srv394035782:84 , srv420682597:84 , srv442754546:84 , srv453417817:84 , srv549365310:84 , srv612416283:84 , srv640441625:84 , srv668585479:84 , srv740316064:84 , srv834132485:84 , srv911737394:84 , srv977737667:84 , srv994033504:84 , srv1066876137:85 , srv1086677687:85 , srv1140002189:85 , srv1201326999:85 , srv1207134992:85 , srv1236121429:85 , srv1289761801:85 , srv1323962104:85 , srv1329889172:85 , srv1343745011:85 , srv1373771051:85 , srv1402833823:85 , srv1418457989:85 , srv1442906609:85 , srv1488533369:85 , srv1597332442:85 , srv1642556172:85 , srv1645288349:85 , srv1759136045:85 , srv1785690214:85 , srv1789765528:85 , srv1844017590:85 , srv1846964592:85 , srv1853938252:85 , srv1943972075:85 , srv1945724048:85 , srv2010705807:85 , srv2017720347:85 , srv2025817003:85 , srv2069338684:85 , srv261849072:85 , srv276550531:85 , srv291222628:85 , srv334989951:85 , srv353771232:85 , srv353876294:85 , srv398783269:85 , srv441656790:85 , srv447210823:85 , srv464337054:85 , srv479478026:85 , srv546375539:85 , srv596931509:85 , srv617160874:85 , srv631378781:85 , srv637325040:85 , srv679371237:85 , srv708959949:85 , srv729047074:85 , srv790884195:85 , srv806016165:85 , srv82590628:85 , srv885983339:85 , srv914385452:85 , srv972640317:85 , srv1158681975:86 , srv1221806104:86 , srv1485015063:86 , srv1827046290:86 , srv2095337228:86 , srv333959821:86 , srv532474436:86 , srv677109922:86 , srv1257620982:619 } 2023-07-28 23:57:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2025817003=47, srv1827046290=39, srv1663087791=34, srv333959821=60, srv994033504=99, srv398783269=66, srv806016165=91, srv885983339=94, srv1257620982=10, srv2095337228=50, srv740316064=89, srv1268852304=11, srv1418457989=20, srv668585479=84, srv273719227=54, srv334989951=61, srv261849072=52, srv164032000=31, srv834132485=93, srv276550531=55, srv82590628=92, srv1402833823=19, srv1010830925=0, srv1294081917=13, srv442754546=69, srv1140002189=4, srv596931509=78, srv1086151291=2, srv1485015063=24, srv1306867584=14, srv1844017590=40, srv1637868355=30, srv394035782=65, srv1158681975=5, srv1759136045=36, srv1488533369=25, srv479478026=74, srv1373771051=18, srv1846964592=41, srv1329889172=16, srv1289761801=12, srv1201326999=6, srv617160874=80, srv1221806104=8, srv1236121429=9, srv1451464229=23, srv464337054=73, srv464030381=72, srv1086677687=3, srv1423619596=21, srv790884195=90, srv1557139706=28, srv1789765528=38, srv532474436=75, srv319801855=58, srv1343745011=17, srv2028817560=48, srv2069338684=49, srv546375539=76, srv453417817=71, srv637325040=82, srv328507217=59, srv1207134992=7, srv631378781=81, srv679371237=86, srv337005277=62, srv1534049159=26, srv914385452=96, srv1668566051=35, srv29746775=57, srv1945724048=44, srv612416283=79, srv1442906609=22, srv1597332442=29, srv267514320=53, srv911737394=95, srv1853938252=42, srv549365310=77, srv1323962104=15, srv640441625=83, srv291222628=56, srv972640317=97, srv353771232=63, srv1642556172=32, srv353876294=64, srv447210823=70, srv677109922=85, srv420682597=67, srv977737667=98, srv1645288349=33, srv2017720347=46, srv441656790=68, srv729047074=88, srv1066876137=1, srv1785690214=37, srv1943972075=43, srv215785232=51, srv708959949=87, srv2010705807=45, srv154687073=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:03,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:03,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:03,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:03,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:03,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:03,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:03,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:03,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 23:57:03,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 23:57:03,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 23:57:03,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 23:57:03,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 23:57:03,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 23:57:03,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 23:57:03,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 23:57:03,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 23:57:03,393 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:03,393 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 23:57:03,393 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6640893882331783E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 23:57:03,574 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 278 ms to try 5000 different iterations. Found a solution that moves 22 regions; Going from a computed imbalance of 2.6640893882331783E-5 to a new imbalance of 7.843469092869584E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0024444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:03,574 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1086677687:83 , srv1485015063:83 , srv1846964592:83 , srv464337054:83 , srv546375539:83 , srv914385452:83 , srv1010830925:84 , srv1066876137:84 , srv1140002189:84 , srv1158681975:84 , srv1201326999:84 , srv1207134992:84 , srv1221806104:84 , srv1236121429:84 , srv1268852304:84 , srv1289761801:84 , srv1294081917:84 , srv1323962104:84 , srv1329889172:84 , srv1343745011:84 , srv1373771051:84 , srv1402833823:84 , srv1418457989:84 , srv1423619596:84 , srv1442906609:84 , srv1451464229:84 , srv1488533369:84 , srv1534049159:84 , srv1557139706:84 , srv1597332442:84 , srv1637868355:84 , srv164032000:84 , srv1642556172:84 , srv1645288349:84 , srv1668566051:84 , srv1759136045:84 , srv1785690214:84 , srv1789765528:84 , srv1827046290:84 , srv1844017590:84 , srv1853938252:84 , srv1943972075:84 , srv1945724048:84 , srv2010705807:84 , srv2025817003:84 , srv2028817560:84 , srv2069338684:84 , srv2095337228:84 , srv215785232:84 , srv261849072:84 , srv267514320:84 , srv273719227:84 , srv276550531:84 , srv291222628:84 , srv29746775:84 , srv319801855:84 , srv328507217:84 , srv333959821:84 , srv334989951:84 , srv337005277:84 , srv353771232:84 , srv353876294:84 , srv394035782:84 , srv398783269:84 , srv420682597:84 , srv441656790:84 , srv442754546:84 , srv447210823:84 , srv453417817:84 , srv464030381:84 , srv479478026:84 , srv532474436:84 , srv549365310:84 , srv596931509:84 , srv612416283:84 , srv617160874:84 , srv631378781:84 , srv637325040:84 , srv640441625:84 , srv668585479:84 , srv677109922:84 , srv679371237:84 , srv708959949:84 , srv729047074:84 , srv740316064:84 , srv790884195:84 , srv806016165:84 , srv82590628:84 , srv834132485:84 , srv885983339:84 , srv911737394:84 , srv972640317:84 , srv994033504:84 , srv1086151291:85 , srv1306867584:85 , srv154687073:85 , srv1663087791:85 , srv2017720347:85 , srv977737667:85 , srv1257620982:684 } 2023-07-28 23:57:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2025817003=47, srv1827046290=39, srv1663087791=34, srv333959821=60, srv994033504=99, srv398783269=66, srv806016165=91, srv885983339=94, srv1257620982=10, srv2095337228=50, srv740316064=89, srv1268852304=11, srv1418457989=20, srv668585479=84, srv273719227=54, srv334989951=61, srv261849072=52, srv164032000=31, srv834132485=93, srv276550531=55, srv82590628=92, srv1402833823=19, srv1010830925=0, srv1294081917=13, srv442754546=69, srv1140002189=4, srv596931509=78, srv1086151291=2, srv1485015063=24, srv1306867584=14, srv1844017590=40, srv1637868355=30, srv394035782=65, srv1158681975=5, srv1759136045=36, srv1488533369=25, srv479478026=74, srv1373771051=18, srv1846964592=41, srv1329889172=16, srv1289761801=12, srv1201326999=6, srv617160874=80, srv1221806104=8, srv1236121429=9, srv1451464229=23, srv464337054=73, srv464030381=72, srv1086677687=3, srv1423619596=21, srv790884195=90, srv1557139706=28, srv1789765528=38, srv532474436=75, srv319801855=58, srv1343745011=17, srv2028817560=48, srv2069338684=49, srv546375539=76, srv453417817=71, srv637325040=82, srv328507217=59, srv1207134992=7, srv631378781=81, srv679371237=86, srv337005277=62, srv1534049159=26, srv914385452=96, srv1668566051=35, srv29746775=57, srv1945724048=44, srv612416283=79, srv1442906609=22, srv1597332442=29, srv267514320=53, srv911737394=95, srv1853938252=42, srv549365310=77, srv1323962104=15, srv640441625=83, srv291222628=56, srv972640317=97, srv353771232=63, srv1642556172=32, srv353876294=64, srv447210823=70, srv677109922=85, srv420682597=67, srv977737667=98, srv1645288349=33, srv2017720347=46, srv441656790=68, srv729047074=88, srv1066876137=1, srv1785690214=37, srv1943972075=43, srv215785232=51, srv708959949=87, srv2010705807=45, srv154687073=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:03,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:03,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 23:57:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 23:57:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 23:57:03,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 23:57:03,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 23:57:03,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 23:57:03,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 23:57:03,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 23:57:03,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 23:57:03,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 23:57:03,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 23:57:03,689 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:03,689 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1086677687:83 , srv1485015063:83 , srv1846964592:83 , srv464337054:83 , srv546375539:83 , srv914385452:83 , srv1010830925:84 , srv1066876137:84 , srv1140002189:84 , srv1158681975:84 , srv1201326999:84 , srv1207134992:84 , srv1221806104:84 , srv1236121429:84 , srv1268852304:84 , srv1289761801:84 , srv1294081917:84 , srv1323962104:84 , srv1329889172:84 , srv1343745011:84 , srv1373771051:84 , srv1402833823:84 , srv1418457989:84 , srv1423619596:84 , srv1442906609:84 , srv1451464229:84 , srv1488533369:84 , srv1534049159:84 , srv1557139706:84 , srv1597332442:84 , srv1637868355:84 , srv164032000:84 , srv1642556172:84 , srv1645288349:84 , srv1668566051:84 , srv1759136045:84 , srv1785690214:84 , srv1789765528:84 , srv1827046290:84 , srv1844017590:84 , srv1853938252:84 , srv1943972075:84 , srv1945724048:84 , srv2010705807:84 , srv2025817003:84 , srv2028817560:84 , srv2069338684:84 , srv2095337228:84 , srv215785232:84 , srv261849072:84 , srv267514320:84 , srv273719227:84 , srv276550531:84 , srv291222628:84 , srv29746775:84 , srv319801855:84 , srv328507217:84 , srv333959821:84 , srv334989951:84 , srv337005277:84 , srv353771232:84 , srv353876294:84 , srv394035782:84 , srv398783269:84 , srv420682597:84 , srv441656790:84 , srv442754546:84 , srv447210823:84 , srv453417817:84 , srv464030381:84 , srv479478026:84 , srv532474436:84 , srv549365310:84 , srv596931509:84 , srv612416283:84 , srv617160874:84 , srv631378781:84 , srv637325040:84 , srv640441625:84 , srv668585479:84 , srv677109922:84 , srv679371237:84 , srv708959949:84 , srv729047074:84 , srv740316064:84 , srv790884195:84 , srv806016165:84 , srv82590628:84 , srv834132485:84 , srv885983339:84 , srv911737394:84 , srv972640317:84 , srv994033504:84 , srv1086151291:85 , srv1306867584:85 , srv154687073:85 , srv1663087791:85 , srv2017720347:85 , srv977737667:85 , srv1257620982:684 } 2023-07-28 23:57:03,722 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=628 (was 630), ProcessCount=191 (was 185) - ProcessCount LEAK? -, AvailableMemoryMB=2457 (was 3189) 2023-07-28 23:57:03,730 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=628, ProcessCount=191, AvailableMemoryMB=2457 2023-07-28 23:57:03,730 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-28 23:57:03,730 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-28 23:57:03,741 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1041167303:84 , srv1075750618:84 , srv1117227175:84 , srv1132114334:84 , srv1200794265:84 , srv1237760855:84 , srv1251625755:84 , srv1254461394:84 , srv1259831701:84 , srv1307118956:84 , srv1320920171:84 , srv1454157392:84 , srv1710358926:84 , srv1711881038:84 , srv1844345319:84 , srv1866572573:84 , srv206736061:84 , srv2095037993:84 , srv2098136990:84 , srv224844209:84 , srv384715440:84 , srv403363161:84 , srv471064893:84 , srv503662504:84 , srv508606319:84 , srv712701603:84 , srv718513174:84 , srv778014820:84 , srv784427862:84 , srv1433919679:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-28 23:57:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:57:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv224844209=20, srv384715440=21, srv2098136990=19, srv1237760855=5, srv1454157392=12, srv1254461394=7, srv471064893=23, srv1711881038=14, srv1307118956=9, srv1251625755=6, srv403363161=22, srv1041167303=0, srv1075750618=1, srv1710358926=13, srv1866572573=16, srv1320920171=10, srv712701603=26, srv206736061=17, srv503662504=24, srv778014820=28, srv1844345319=15, srv508606319=25, srv1200794265=4, srv1117227175=2, srv1132114334=3, srv1433919679=11, srv784427862=29, srv1259831701=8, srv718513174=27, srv2095037993=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:57:03,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:57:03,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:57:03,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:57:03,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:57:03,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:57:03,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:57:03,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:57:03,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:57:03,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-28 23:57:03,790 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1396208194344752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:57:03,791 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912632983907278, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1396208194344752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-28 23:59:03,776 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 8267240 different iterations. Found a solution that moves 1841 regions; Going from a computed imbalance of 0.9912632983907278 to a new imbalance of 4.296985841319951E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6818518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:59:03,782 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1041167303:90 , srv1075750618:90 , srv1117227175:90 , srv1132114334:90 , srv1200794265:90 , srv1237760855:90 , srv1251625755:90 , srv1254461394:90 , srv1259831701:90 , srv1307118956:90 , srv1320920171:90 , srv1433919679:90 , srv1454157392:90 , srv1710358926:90 , srv1711881038:90 , srv1844345319:90 , srv1866572573:90 , srv206736061:90 , srv2095037993:90 , srv2098136990:90 , srv224844209:90 , srv384715440:90 , srv403363161:90 , srv471064893:90 , srv503662504:90 , srv508606319:90 , srv712701603:90 , srv718513174:90 , srv778014820:90 , srv784427862:90 } 2023-07-28 23:59:03,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 23:59:03,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv224844209=20, srv384715440=21, srv2098136990=19, srv1237760855=5, srv1454157392=12, srv1254461394=7, srv471064893=23, srv1711881038=14, srv1307118956=9, srv1251625755=6, srv403363161=22, srv1041167303=0, srv1075750618=1, srv1710358926=13, srv1866572573=16, srv1320920171=10, srv712701603=26, srv206736061=17, srv503662504=24, srv778014820=28, srv1844345319=15, srv508606319=25, srv1200794265=4, srv1117227175=2, srv1132114334=3, srv1433919679=11, srv784427862=29, srv1259831701=8, srv718513174=27, srv2095037993=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 23:59:03,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 23:59:03,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 23:59:03,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 23:59:03,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-28 23:59:03,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 23:59:03,799 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1041167303:90 , srv1075750618:90 , srv1117227175:90 , srv1132114334:90 , srv1200794265:90 , srv1237760855:90 , srv1251625755:90 , srv1254461394:90 , srv1259831701:90 , srv1307118956:90 , srv1320920171:90 , srv1433919679:90 , srv1454157392:90 , srv1710358926:90 , srv1711881038:90 , srv1844345319:90 , srv1866572573:90 , srv206736061:90 , srv2095037993:90 , srv2098136990:90 , srv224844209:90 , srv384715440:90 , srv403363161:90 , srv471064893:90 , srv503662504:90 , srv508606319:90 , srv712701603:90 , srv718513174:90 , srv778014820:90 , srv784427862:90 } 2023-07-28 23:59:03,812 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=462 (was 628), ProcessCount=178 (was 191), AvailableMemoryMB=4723 (was 2457) - AvailableMemoryMB LEAK? -