2023-09-04 02:53:13,712 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-04 02:53:14,049 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-04 02:53:14,171 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-04 02:53:14,238 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-04 02:53:14,248 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-04 02:53:14,264 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=169, AvailableMemoryMB=10217 2023-09-04 02:53:14,266 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-04 02:53:14,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-04 02:53:14,407 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003243540:84 , srv1012561083:84 , srv1027512831:84 , srv1028498112:84 , srv105303874:84 , srv1066650587:84 , srv1071858972:84 , srv1089173439:84 , srv1125839123:84 , srv115525682:84 , srv118854687:84 , srv1202736375:84 , srv1205438000:84 , srv1206913067:84 , srv1225439914:84 , srv1236184880:84 , srv1236455151:84 , srv1282707559:84 , srv1304995528:84 , srv133329277:84 , srv136141927:84 , srv1366724287:84 , srv1374379814:84 , srv1436454412:84 , srv1471599768:84 , srv1475313683:84 , srv1495154858:84 , srv1515206677:84 , srv1537812623:84 , srv1547531364:84 , srv1557960443:84 , srv1558475486:84 , srv1565574002:84 , srv1571221536:84 , srv1601516001:84 , srv1638691337:84 , srv1639611737:84 , srv1643583930:84 , srv1659225469:84 , srv1698127607:84 , srv1733534558:84 , srv1734907439:84 , srv1742638656:84 , srv1782373179:84 , srv1802438246:84 , srv181412324:84 , srv1821085716:84 , srv1825541095:84 , srv1869145965:84 , srv1896384385:84 , srv1935850308:84 , srv1939485335:84 , srv1943160495:84 , srv195104837:84 , srv1959869447:84 , srv1968499855:84 , srv1970371117:84 , srv1972225150:84 , srv1993423961:84 , srv2012878151:84 , srv2023925627:84 , srv2044657013:84 , srv2057897162:84 , srv2066751536:84 , srv2083196887:84 , srv2088534868:84 , srv2093752203:84 , srv2136892019:84 , srv214564194:84 , srv225931784:84 , srv261751622:84 , srv267270941:84 , srv345184466:84 , srv407142273:84 , srv421233417:84 , srv440382676:84 , srv442654670:84 , srv480798893:84 , srv488845320:84 , srv499819714:84 , srv516019949:84 , srv575256418:84 , srv609015716:84 , srv661728708:84 , srv682424145:84 , srv687169442:84 , srv693421175:84 , srv699863096:84 , srv742494029:84 , srv748343518:84 , srv76515136:84 , srv780909536:84 , srv783613215:84 , srv785013162:84 , srv797525591:84 , srv912911367:84 , srv912986594:84 , srv933103470:84 , srv985939102:84 , srv1938852547:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-04 02:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv105303874=4, srv195104837=54, srv912986594=97, srv1495154858=26, srv687169442=86, srv407142273=74, srv661728708=84, srv1733534558=40, srv2136892019=68, srv1896384385=49, srv1659225469=38, srv115525682=9, srv2083196887=65, srv1601516001=34, srv76515136=91, srv1643583930=37, srv1003243540=0, srv440382676=76, srv912911367=96, srv133329277=19, srv1968499855=56, srv1821085716=46, srv609015716=83, srv1515206677=27, srv1638691337=35, srv742494029=89, srv1374379814=22, srv1972225150=58, srv985939102=99, srv1571221536=33, srv214564194=69, srv1236455151=16, srv1742638656=42, srv1869145965=48, srv1993423961=59, srv2093752203=67, srv1734907439=41, srv2057897162=63, srv499819714=80, srv1066650587=5, srv267270941=72, srv1639611737=36, srv682424145=85, srv1027512831=2, srv1028498112=3, srv1206913067=13, srv225931784=70, srv442654670=77, srv1782373179=43, srv1970371117=57, srv1071858972=6, srv699863096=88, srv1225439914=14, srv797525591=95, srv1475313683=25, srv1125839123=8, srv1825541095=47, srv1282707559=17, srv780909536=92, srv421233417=75, srv1943160495=53, srv345184466=73, srv1557960443=30, srv1236184880=15, srv783613215=93, srv1436454412=23, srv516019949=81, srv1366724287=21, srv785013162=94, srv1565574002=32, srv575256418=82, srv2023925627=61, srv693421175=87, srv1547531364=29, srv1205438000=12, srv1089173439=7, srv748343518=90, srv1935850308=50, srv1471599768=24, srv1202736375=11, srv261751622=71, srv933103470=98, srv1698127607=39, srv2044657013=62, srv136141927=20, srv2088534868=66, srv118854687=10, srv2066751536=64, srv181412324=45, srv1012561083=1, srv1802438246=44, srv1304995528=18, srv1558475486=31, srv1959869447=55, srv1537812623=28, srv2012878151=60, srv480798893=78, srv488845320=79, srv1938852547=51, srv1939485335=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:14,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:14,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:14,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:14,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:14,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:14,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:14,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:14,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:14,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:14,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:14,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:14,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:14,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:14,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:14,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:14,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:14,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:14,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-04 02:53:14,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-04 02:53:14,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-04 02:53:14,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-04 02:53:14,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-04 02:53:14,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-04 02:53:14,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-04 02:53:14,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-04 02:53:14,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-04 02:53:14,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-04 02:53:14,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-04 02:53:14,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-04 02:53:14,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-04 02:53:14,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-04 02:53:14,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-04 02:53:14,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-04 02:53:14,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-04 02:53:14,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-04 02:53:14,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-04 02:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-04 02:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-04 02:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-04 02:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-04 02:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-04 02:53:14,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-04 02:53:14,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-04 02:53:14,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-04 02:53:14,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-04 02:53:14,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-04 02:53:14,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-04 02:53:14,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-04 02:53:14,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-04 02:53:14,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-04 02:53:14,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-04 02:53:14,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-04 02:53:14,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-04 02:53:14,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-04 02:53:14,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-04 02:53:14,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-04 02:53:14,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-04 02:53:14,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-04 02:53:14,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-04 02:53:14,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-04 02:53:14,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-04 02:53:14,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-04 02:53:14,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-04 02:53:14,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-04 02:53:14,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:14,787 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-04 02:53:14,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-04 02:53:15,140 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 576 ms to try 5000 different iterations. Found a solution that moves 4687 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18075600036884837. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04074104683080607, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05645136360995295, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5207777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04074104683080607, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:15,161 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1639611737:78 , srv1495154858:80 , srv499819714:80 , srv748343518:81 , srv912986594:81 , srv1012561083:82 , srv1557960443:83 , srv1558475486:83 , srv1601516001:83 , srv1659225469:83 , srv1733534558:83 , srv1993423961:83 , srv105303874:84 , srv1071858972:84 , srv1236184880:84 , srv136141927:84 , srv1436454412:84 , srv1734907439:84 , srv1802438246:84 , srv1943160495:84 , srv2012878151:84 , srv2044657013:84 , srv214564194:84 , srv742494029:84 , srv1066650587:85 , srv1236455151:85 , srv1366724287:85 , srv1374379814:85 , srv1537812623:85 , srv1643583930:85 , srv1782373179:85 , srv1869145965:85 , srv195104837:85 , srv2088534868:85 , srv440382676:85 , srv609015716:85 , srv1206913067:86 , srv1282707559:86 , srv1515206677:86 , srv1565574002:86 , srv1698127607:86 , srv1935850308:86 , srv1939485335:86 , srv2083196887:86 , srv267270941:86 , srv488845320:86 , srv780909536:86 , srv912911367:86 , srv985939102:86 , srv1003243540:87 , srv1027512831:87 , srv1089173439:87 , srv115525682:87 , srv1304995528:87 , srv1475313683:87 , srv1547531364:87 , srv1742638656:87 , srv1959869447:87 , srv2057897162:87 , srv225931784:87 , srv345184466:87 , srv421233417:87 , srv480798893:87 , srv516019949:87 , srv661728708:87 , srv693421175:87 , srv699863096:87 , srv785013162:87 , srv1028498112:88 , srv1125839123:88 , srv118854687:88 , srv1205438000:88 , srv133329277:88 , srv1571221536:88 , srv181412324:88 , srv1821085716:88 , srv1968499855:88 , srv1970371117:88 , srv2066751536:88 , srv2136892019:88 , srv1225439914:89 , srv1471599768:89 , srv1825541095:89 , srv2093752203:89 , srv407142273:89 , srv575256418:89 , srv682424145:89 , srv687169442:89 , srv783613215:89 , srv1202736375:90 , srv1638691337:90 , srv1896384385:90 , srv1972225150:91 , srv2023925627:91 , srv76515136:91 , srv933103470:91 , srv261751622:92 , srv442654670:93 , srv797525591:93 , srv1938852547:452 } 2023-09-04 02:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv105303874=4, srv195104837=54, srv912986594=97, srv1495154858=26, srv687169442=86, srv407142273=74, srv661728708=84, srv1733534558=40, srv2136892019=68, srv1896384385=49, srv1659225469=38, srv115525682=9, srv2083196887=65, srv1601516001=34, srv76515136=91, srv1643583930=37, srv1003243540=0, srv440382676=76, srv912911367=96, srv133329277=19, srv1968499855=56, srv1821085716=46, srv609015716=83, srv1515206677=27, srv1638691337=35, srv742494029=89, srv1374379814=22, srv1972225150=58, srv985939102=99, srv1571221536=33, srv214564194=69, srv1236455151=16, srv1742638656=42, srv1869145965=48, srv1993423961=59, srv2093752203=67, srv1734907439=41, srv2057897162=63, srv499819714=80, srv1066650587=5, srv267270941=72, srv1639611737=36, srv682424145=85, srv1027512831=2, srv1028498112=3, srv1206913067=13, srv225931784=70, srv442654670=77, srv1782373179=43, srv1970371117=57, srv1071858972=6, srv699863096=88, srv1225439914=14, srv797525591=95, srv1475313683=25, srv1125839123=8, srv1825541095=47, srv1282707559=17, srv780909536=92, srv421233417=75, srv1943160495=53, srv345184466=73, srv1557960443=30, srv1236184880=15, srv783613215=93, srv1436454412=23, srv516019949=81, srv1366724287=21, srv785013162=94, srv1565574002=32, srv575256418=82, srv2023925627=61, srv693421175=87, srv1547531364=29, srv1205438000=12, srv1089173439=7, srv748343518=90, srv1935850308=50, srv1471599768=24, srv1202736375=11, srv261751622=71, srv933103470=98, srv1698127607=39, srv2044657013=62, srv136141927=20, srv2088534868=66, srv118854687=10, srv2066751536=64, srv181412324=45, srv1012561083=1, srv1802438246=44, srv1304995528=18, srv1558475486=31, srv1959869447=55, srv1537812623=28, srv2012878151=60, srv480798893=78, srv488845320=79, srv1938852547=51, srv1939485335=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-04 02:53:15,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-04 02:53:15,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-04 02:53:15,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-04 02:53:15,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-04 02:53:15,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-04 02:53:15,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-04 02:53:15,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-04 02:53:15,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-04 02:53:15,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-04 02:53:15,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-04 02:53:15,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-04 02:53:15,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-04 02:53:15,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-04 02:53:15,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-04 02:53:15,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-04 02:53:15,401 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04074104683080607, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05645136360995295, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04074104683080607, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:15,402 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-04 02:53:15,402 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1807231812933922, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04074104683080607, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05645136360995295, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04074104683080607, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-04 02:53:15,739 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 497 ms to try 5000 different iterations. Found a solution that moves 2654 regions; Going from a computed imbalance of 0.1807231812933922 to a new imbalance of 0.01399302485913488. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020830733096241446); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03218911569882749, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29488888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020830733096241446); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:15,751 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv797525591:79 , srv1972225150:80 , srv1821085716:81 , srv261751622:81 , srv682424145:81 , srv1638691337:82 , srv2023925627:82 , srv442654670:82 , srv575256418:82 , srv76515136:82 , srv1003243540:83 , srv1028498112:83 , srv1089173439:83 , srv1202736375:83 , srv133329277:83 , srv1825541095:83 , srv1968499855:83 , srv1970371117:83 , srv687169442:83 , srv783613215:83 , srv785013162:83 , srv933103470:83 , srv1125839123:84 , srv118854687:84 , srv1225439914:84 , srv1471599768:84 , srv1475313683:84 , srv1547531364:84 , srv1742638656:84 , srv1869145965:84 , srv2066751536:84 , srv2083196887:84 , srv2093752203:84 , srv480798893:84 , srv1027512831:85 , srv1205438000:85 , srv1565574002:85 , srv1571221536:85 , srv1896384385:85 , srv2088534868:85 , srv2136892019:85 , srv225931784:85 , srv267270941:85 , srv345184466:85 , srv407142273:85 , srv421233417:85 , srv440382676:85 , srv516019949:85 , srv661728708:85 , srv693421175:85 , srv912911367:85 , srv105303874:86 , srv1206913067:86 , srv1282707559:86 , srv1515206677:86 , srv1698127607:86 , srv1782373179:86 , srv181412324:86 , srv1935850308:86 , srv1943160495:86 , srv2012878151:86 , srv2057897162:86 , srv609015716:86 , srv985939102:86 , srv1066650587:87 , srv1236455151:87 , srv1304995528:87 , srv136141927:87 , srv1537812623:87 , srv1557960443:87 , srv1659225469:87 , srv780909536:87 , srv1071858972:88 , srv115525682:88 , srv1236184880:88 , srv1374379814:88 , srv1601516001:88 , srv1959869447:88 , srv2044657013:88 , srv488845320:88 , srv699863096:88 , srv1436454412:89 , srv1733534558:89 , srv1734907439:89 , srv1939485335:89 , srv214564194:89 , srv1012561083:90 , srv1366724287:90 , srv1558475486:90 , srv1802438246:90 , srv1993423961:90 , srv742494029:90 , srv748343518:90 , srv1643583930:91 , srv195104837:91 , srv1495154858:92 , srv499819714:92 , srv912986594:92 , srv1639611737:93 , srv1938852547:507 } 2023-09-04 02:53:15,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:15,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv105303874=4, srv195104837=54, srv912986594=97, srv1495154858=26, srv687169442=86, srv407142273=74, srv661728708=84, srv1733534558=40, srv2136892019=68, srv1896384385=49, srv1659225469=38, srv115525682=9, srv2083196887=65, srv1601516001=34, srv76515136=91, srv1643583930=37, srv1003243540=0, srv440382676=76, srv912911367=96, srv133329277=19, srv1968499855=56, srv1821085716=46, srv609015716=83, srv1515206677=27, srv1638691337=35, srv742494029=89, srv1374379814=22, srv1972225150=58, srv985939102=99, srv1571221536=33, srv214564194=69, srv1236455151=16, srv1742638656=42, srv1869145965=48, srv1993423961=59, srv2093752203=67, srv1734907439=41, srv2057897162=63, srv499819714=80, srv1066650587=5, srv267270941=72, srv1639611737=36, srv682424145=85, srv1027512831=2, srv1028498112=3, srv1206913067=13, srv225931784=70, srv442654670=77, srv1782373179=43, srv1970371117=57, srv1071858972=6, srv699863096=88, srv1225439914=14, srv797525591=95, srv1475313683=25, srv1125839123=8, srv1825541095=47, srv1282707559=17, srv780909536=92, srv421233417=75, srv1943160495=53, srv345184466=73, srv1557960443=30, srv1236184880=15, srv783613215=93, srv1436454412=23, srv516019949=81, srv1366724287=21, srv785013162=94, srv1565574002=32, srv575256418=82, srv2023925627=61, srv693421175=87, srv1547531364=29, srv1205438000=12, srv1089173439=7, srv748343518=90, srv1935850308=50, srv1471599768=24, srv1202736375=11, srv261751622=71, srv933103470=98, srv1698127607=39, srv2044657013=62, srv136141927=20, srv2088534868=66, srv118854687=10, srv2066751536=64, srv181412324=45, srv1012561083=1, srv1802438246=44, srv1304995528=18, srv1558475486=31, srv1959869447=55, srv1537812623=28, srv2012878151=60, srv480798893=78, srv488845320=79, srv1938852547=51, srv1939485335=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-04 02:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-04 02:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-04 02:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-04 02:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-04 02:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-04 02:53:15,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-04 02:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-04 02:53:15,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-04 02:53:15,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-04 02:53:15,961 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020830733096241446); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03218911569882749, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020830733096241446); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:15,962 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-04 02:53:15,962 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013974441153937386, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020830733096241446); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03218911569882749, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020830733096241446); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-04 02:53:16,322 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 515 ms to try 5000 different iterations. Found a solution that moves 999 regions; Going from a computed imbalance of 0.013974441153937386 to a new imbalance of 6.636943239942017E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054249519356559806); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054249519356559806); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:16,325 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv195104837:81 , srv1959869447:82 , srv115525682:83 , srv1366724287:83 , srv1643583930:83 , srv181412324:83 , srv1896384385:83 , srv1939485335:83 , srv407142273:83 , srv442654670:83 , srv699863096:83 , srv742494029:83 , srv1202736375:84 , srv1304995528:84 , srv1374379814:84 , srv1802438246:84 , srv2057897162:84 , srv2136892019:84 , srv214564194:84 , srv261751622:84 , srv488845320:84 , srv780909536:84 , srv912986594:84 , srv933103470:84 , srv1027512831:85 , srv1066650587:85 , srv1071858972:85 , srv118854687:85 , srv1205438000:85 , srv1206913067:85 , srv1225439914:85 , srv1236184880:85 , srv1436454412:85 , srv1471599768:85 , srv1515206677:85 , srv1537812623:85 , srv1558475486:85 , srv1571221536:85 , srv1698127607:85 , srv1733534558:85 , srv1734907439:85 , srv1993423961:85 , srv2023925627:85 , srv2044657013:85 , srv2093752203:85 , srv345184466:85 , srv421233417:85 , srv661728708:85 , srv687169442:85 , srv693421175:85 , srv76515136:85 , srv783613215:85 , srv797525591:85 , srv985939102:85 , srv1012561083:86 , srv1028498112:86 , srv1125839123:86 , srv1236455151:86 , srv1282707559:86 , srv133329277:86 , srv136141927:86 , srv1475313683:86 , srv1495154858:86 , srv1547531364:86 , srv1565574002:86 , srv1601516001:86 , srv1638691337:86 , srv1742638656:86 , srv1825541095:86 , srv1935850308:86 , srv1968499855:86 , srv1970371117:86 , srv2066751536:86 , srv225931784:86 , srv480798893:86 , srv499819714:86 , srv575256418:86 , srv609015716:86 , srv748343518:86 , srv1003243540:87 , srv1089173439:87 , srv1639611737:87 , srv1659225469:87 , srv1782373179:87 , srv1943160495:87 , srv1972225150:87 , srv267270941:87 , srv516019949:87 , srv912911367:87 , srv105303874:88 , srv1557960443:88 , srv1821085716:88 , srv1869145965:88 , srv2012878151:88 , srv2088534868:88 , srv440382676:88 , srv785013162:88 , srv2083196887:89 , srv682424145:89 , srv1938852547:547 } 2023-09-04 02:53:16,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:16,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv105303874=4, srv195104837=54, srv912986594=97, srv1495154858=26, srv687169442=86, srv407142273=74, srv661728708=84, srv1733534558=40, srv2136892019=68, srv1896384385=49, srv1659225469=38, srv115525682=9, srv2083196887=65, srv1601516001=34, srv76515136=91, srv1643583930=37, srv1003243540=0, srv440382676=76, srv912911367=96, srv133329277=19, srv1968499855=56, srv1821085716=46, srv609015716=83, srv1515206677=27, srv1638691337=35, srv742494029=89, srv1374379814=22, srv1972225150=58, srv985939102=99, srv1571221536=33, srv214564194=69, srv1236455151=16, srv1742638656=42, srv1869145965=48, srv1993423961=59, srv2093752203=67, srv1734907439=41, srv2057897162=63, srv499819714=80, srv1066650587=5, srv267270941=72, srv1639611737=36, srv682424145=85, srv1027512831=2, srv1028498112=3, srv1206913067=13, srv225931784=70, srv442654670=77, srv1782373179=43, srv1970371117=57, srv1071858972=6, srv699863096=88, srv1225439914=14, srv797525591=95, srv1475313683=25, srv1125839123=8, srv1825541095=47, srv1282707559=17, srv780909536=92, srv421233417=75, srv1943160495=53, srv345184466=73, srv1557960443=30, srv1236184880=15, srv783613215=93, srv1436454412=23, srv516019949=81, srv1366724287=21, srv785013162=94, srv1565574002=32, srv575256418=82, srv2023925627=61, srv693421175=87, srv1547531364=29, srv1205438000=12, srv1089173439=7, srv748343518=90, srv1935850308=50, srv1471599768=24, srv1202736375=11, srv261751622=71, srv933103470=98, srv1698127607=39, srv2044657013=62, srv136141927=20, srv2088534868=66, srv118854687=10, srv2066751536=64, srv181412324=45, srv1012561083=1, srv1802438246=44, srv1304995528=18, srv1558475486=31, srv1959869447=55, srv1537812623=28, srv2012878151=60, srv480798893=78, srv488845320=79, srv1938852547=51, srv1939485335=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:16,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:16,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:16,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-04 02:53:16,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-04 02:53:16,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-04 02:53:16,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-04 02:53:16,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-04 02:53:16,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-04 02:53:16,496 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054249519356559806); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054249519356559806); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:16,497 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-04 02:53:16,497 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.566991764839162E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054249519356559806); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054249519356559806); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-04 02:53:16,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 559 ms to try 5000 different iterations. Found a solution that moves 222 regions; Going from a computed imbalance of 6.566991764839162E-4 to a new imbalance of 6.297965598981887E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.024666666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:16,932 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1643583930:83 , srv1825541095:83 , srv2083196887:83 , srv516019949:83 , srv682424145:83 , srv1012561083:84 , srv105303874:84 , srv1125839123:84 , srv115525682:84 , srv1205438000:84 , srv1225439914:84 , srv1236184880:84 , srv1236455151:84 , srv1282707559:84 , srv1304995528:84 , srv1436454412:84 , srv1471599768:84 , srv1495154858:84 , srv1515206677:84 , srv1557960443:84 , srv1558475486:84 , srv1571221536:84 , srv1638691337:84 , srv1639611737:84 , srv1734907439:84 , srv1782373179:84 , srv1802438246:84 , srv1896384385:84 , srv1935850308:84 , srv1939485335:84 , srv1972225150:84 , srv1993423961:84 , srv2023925627:84 , srv2066751536:84 , srv2088534868:84 , srv2093752203:84 , srv225931784:84 , srv267270941:84 , srv440382676:84 , srv442654670:84 , srv488845320:84 , srv499819714:84 , srv609015716:84 , srv699863096:84 , srv76515136:84 , srv785013162:84 , srv912911367:84 , srv1003243540:85 , srv1027512831:85 , srv1028498112:85 , srv1066650587:85 , srv1089173439:85 , srv118854687:85 , srv1202736375:85 , srv1206913067:85 , srv133329277:85 , srv136141927:85 , srv1366724287:85 , srv1374379814:85 , srv1475313683:85 , srv1537812623:85 , srv1547531364:85 , srv1565574002:85 , srv1601516001:85 , srv1659225469:85 , srv1698127607:85 , srv1733534558:85 , srv1742638656:85 , srv181412324:85 , srv1821085716:85 , srv1869145965:85 , srv1943160495:85 , srv195104837:85 , srv1959869447:85 , srv1968499855:85 , srv1970371117:85 , srv2012878151:85 , srv2044657013:85 , srv2057897162:85 , srv2136892019:85 , srv214564194:85 , srv261751622:85 , srv345184466:85 , srv407142273:85 , srv421233417:85 , srv480798893:85 , srv575256418:85 , srv661728708:85 , srv687169442:85 , srv693421175:85 , srv742494029:85 , srv748343518:85 , srv780909536:85 , srv783613215:85 , srv797525591:85 , srv912986594:85 , srv933103470:85 , srv985939102:85 , srv1071858972:86 , srv1938852547:636 } 2023-09-04 02:53:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv105303874=4, srv195104837=54, srv912986594=97, srv1495154858=26, srv687169442=86, srv407142273=74, srv661728708=84, srv1733534558=40, srv2136892019=68, srv1896384385=49, srv1659225469=38, srv115525682=9, srv2083196887=65, srv1601516001=34, srv76515136=91, srv1643583930=37, srv1003243540=0, srv440382676=76, srv912911367=96, srv133329277=19, srv1968499855=56, srv1821085716=46, srv609015716=83, srv1515206677=27, srv1638691337=35, srv742494029=89, srv1374379814=22, srv1972225150=58, srv985939102=99, srv1571221536=33, srv214564194=69, srv1236455151=16, srv1742638656=42, srv1869145965=48, srv1993423961=59, srv2093752203=67, srv1734907439=41, srv2057897162=63, srv499819714=80, srv1066650587=5, srv267270941=72, srv1639611737=36, srv682424145=85, srv1027512831=2, srv1028498112=3, srv1206913067=13, srv225931784=70, srv442654670=77, srv1782373179=43, srv1970371117=57, srv1071858972=6, srv699863096=88, srv1225439914=14, srv797525591=95, srv1475313683=25, srv1125839123=8, srv1825541095=47, srv1282707559=17, srv780909536=92, srv421233417=75, srv1943160495=53, srv345184466=73, srv1557960443=30, srv1236184880=15, srv783613215=93, srv1436454412=23, srv516019949=81, srv1366724287=21, srv785013162=94, srv1565574002=32, srv575256418=82, srv2023925627=61, srv693421175=87, srv1547531364=29, srv1205438000=12, srv1089173439=7, srv748343518=90, srv1935850308=50, srv1471599768=24, srv1202736375=11, srv261751622=71, srv933103470=98, srv1698127607=39, srv2044657013=62, srv136141927=20, srv2088534868=66, srv118854687=10, srv2066751536=64, srv181412324=45, srv1012561083=1, srv1802438246=44, srv1304995528=18, srv1558475486=31, srv1959869447=55, srv1537812623=28, srv2012878151=60, srv480798893=78, srv488845320=79, srv1938852547=51, srv1939485335=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-04 02:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-04 02:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-04 02:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-04 02:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-04 02:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-04 02:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-04 02:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-04 02:53:17,098 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:17,098 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-04 02:53:17,099 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.142517876531093E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-04 02:53:17,405 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 423 ms to try 5000 different iterations. Found a solution that moves 29 regions; Going from a computed imbalance of 6.142517876531093E-5 to a new imbalance of 6.601048012408121E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0032222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:17,405 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1071858972:83 , srv1366724287:83 , srv2012878151:83 , srv933103470:83 , srv1003243540:84 , srv1012561083:84 , srv1027512831:84 , srv1028498112:84 , srv105303874:84 , srv1066650587:84 , srv1089173439:84 , srv1125839123:84 , srv115525682:84 , srv118854687:84 , srv1202736375:84 , srv1205438000:84 , srv1206913067:84 , srv1225439914:84 , srv1236455151:84 , srv1282707559:84 , srv1304995528:84 , srv133329277:84 , srv136141927:84 , srv1374379814:84 , srv1436454412:84 , srv1471599768:84 , srv1475313683:84 , srv1495154858:84 , srv1537812623:84 , srv1547531364:84 , srv1557960443:84 , srv1558475486:84 , srv1565574002:84 , srv1571221536:84 , srv1601516001:84 , srv1638691337:84 , srv1639611737:84 , srv1643583930:84 , srv1659225469:84 , srv1698127607:84 , srv1733534558:84 , srv1734907439:84 , srv1742638656:84 , srv1782373179:84 , srv1802438246:84 , srv181412324:84 , srv1821085716:84 , srv1869145965:84 , srv1896384385:84 , srv1935850308:84 , srv1939485335:84 , srv1943160495:84 , srv195104837:84 , srv1959869447:84 , srv1968499855:84 , srv1970371117:84 , srv1972225150:84 , srv1993423961:84 , srv2023925627:84 , srv2044657013:84 , srv2057897162:84 , srv2066751536:84 , srv2083196887:84 , srv2088534868:84 , srv2093752203:84 , srv2136892019:84 , srv214564194:84 , srv225931784:84 , srv261751622:84 , srv267270941:84 , srv345184466:84 , srv407142273:84 , srv421233417:84 , srv440382676:84 , srv442654670:84 , srv480798893:84 , srv488845320:84 , srv499819714:84 , srv516019949:84 , srv575256418:84 , srv661728708:84 , srv682424145:84 , srv687169442:84 , srv693421175:84 , srv699863096:84 , srv742494029:84 , srv748343518:84 , srv76515136:84 , srv780909536:84 , srv783613215:84 , srv785013162:84 , srv797525591:84 , srv912911367:84 , srv912986594:84 , srv985939102:84 , srv1236184880:85 , srv1515206677:85 , srv1825541095:85 , srv609015716:85 , srv1938852547:684 } 2023-09-04 02:53:17,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:17,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv105303874=4, srv195104837=54, srv912986594=97, srv1495154858=26, srv687169442=86, srv407142273=74, srv661728708=84, srv1733534558=40, srv2136892019=68, srv1896384385=49, srv1659225469=38, srv115525682=9, srv2083196887=65, srv1601516001=34, srv76515136=91, srv1643583930=37, srv1003243540=0, srv440382676=76, srv912911367=96, srv133329277=19, srv1968499855=56, srv1821085716=46, srv609015716=83, srv1515206677=27, srv1638691337=35, srv742494029=89, srv1374379814=22, srv1972225150=58, srv985939102=99, srv1571221536=33, srv214564194=69, srv1236455151=16, srv1742638656=42, srv1869145965=48, srv1993423961=59, srv2093752203=67, srv1734907439=41, srv2057897162=63, srv499819714=80, srv1066650587=5, srv267270941=72, srv1639611737=36, srv682424145=85, srv1027512831=2, srv1028498112=3, srv1206913067=13, srv225931784=70, srv442654670=77, srv1782373179=43, srv1970371117=57, srv1071858972=6, srv699863096=88, srv1225439914=14, srv797525591=95, srv1475313683=25, srv1125839123=8, srv1825541095=47, srv1282707559=17, srv780909536=92, srv421233417=75, srv1943160495=53, srv345184466=73, srv1557960443=30, srv1236184880=15, srv783613215=93, srv1436454412=23, srv516019949=81, srv1366724287=21, srv785013162=94, srv1565574002=32, srv575256418=82, srv2023925627=61, srv693421175=87, srv1547531364=29, srv1205438000=12, srv1089173439=7, srv748343518=90, srv1935850308=50, srv1471599768=24, srv1202736375=11, srv261751622=71, srv933103470=98, srv1698127607=39, srv2044657013=62, srv136141927=20, srv2088534868=66, srv118854687=10, srv2066751536=64, srv181412324=45, srv1012561083=1, srv1802438246=44, srv1304995528=18, srv1558475486=31, srv1959869447=55, srv1537812623=28, srv2012878151=60, srv480798893=78, srv488845320=79, srv1938852547=51, srv1939485335=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-04 02:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-04 02:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-04 02:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-04 02:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-04 02:53:17,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-04 02:53:17,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-04 02:53:17,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-04 02:53:17,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-04 02:53:17,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-04 02:53:17,581 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:17,581 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1071858972:83 , srv1366724287:83 , srv2012878151:83 , srv933103470:83 , srv1003243540:84 , srv1012561083:84 , srv1027512831:84 , srv1028498112:84 , srv105303874:84 , srv1066650587:84 , srv1089173439:84 , srv1125839123:84 , srv115525682:84 , srv118854687:84 , srv1202736375:84 , srv1205438000:84 , srv1206913067:84 , srv1225439914:84 , srv1236455151:84 , srv1282707559:84 , srv1304995528:84 , srv133329277:84 , srv136141927:84 , srv1374379814:84 , srv1436454412:84 , srv1471599768:84 , srv1475313683:84 , srv1495154858:84 , srv1537812623:84 , srv1547531364:84 , srv1557960443:84 , srv1558475486:84 , srv1565574002:84 , srv1571221536:84 , srv1601516001:84 , srv1638691337:84 , srv1639611737:84 , srv1643583930:84 , srv1659225469:84 , srv1698127607:84 , srv1733534558:84 , srv1734907439:84 , srv1742638656:84 , srv1782373179:84 , srv1802438246:84 , srv181412324:84 , srv1821085716:84 , srv1869145965:84 , srv1896384385:84 , srv1935850308:84 , srv1939485335:84 , srv1943160495:84 , srv195104837:84 , srv1959869447:84 , srv1968499855:84 , srv1970371117:84 , srv1972225150:84 , srv1993423961:84 , srv2023925627:84 , srv2044657013:84 , srv2057897162:84 , srv2066751536:84 , srv2083196887:84 , srv2088534868:84 , srv2093752203:84 , srv2136892019:84 , srv214564194:84 , srv225931784:84 , srv261751622:84 , srv267270941:84 , srv345184466:84 , srv407142273:84 , srv421233417:84 , srv440382676:84 , srv442654670:84 , srv480798893:84 , srv488845320:84 , srv499819714:84 , srv516019949:84 , srv575256418:84 , srv661728708:84 , srv682424145:84 , srv687169442:84 , srv693421175:84 , srv699863096:84 , srv742494029:84 , srv748343518:84 , srv76515136:84 , srv780909536:84 , srv783613215:84 , srv785013162:84 , srv797525591:84 , srv912911367:84 , srv912986594:84 , srv985939102:84 , srv1236184880:85 , srv1515206677:85 , srv1825541095:85 , srv609015716:85 , srv1938852547:684 } 2023-09-04 02:53:17,618 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=361 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=9347 (was 10217) 2023-09-04 02:53:17,628 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=361, ProcessCount=169, AvailableMemoryMB=9346 2023-09-04 02:53:17,629 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-04 02:53:17,629 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-04 02:53:17,641 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1112494275:84 , srv1409701346:84 , srv1425259299:84 , srv147417365:84 , srv1609569731:84 , srv1687121652:84 , srv1692649306:84 , srv171616298:84 , srv1726647849:84 , srv1883439434:84 , srv1908517245:84 , srv2124294864:84 , srv2130286189:84 , srv262375415:84 , srv304045273:84 , srv313738644:84 , srv351557284:84 , srv432507554:84 , srv543076232:84 , srv559285325:84 , srv566777636:84 , srv63231189:84 , srv657946362:84 , srv714706804:84 , srv717558130:84 , srv819493842:84 , srv870615803:84 , srv96596303:84 , srv984408987:84 , srv1645578510:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-04 02:53:17,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:53:17,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1645578510=5, srv1692649306=7, srv262375415=14, srv313738644=16, srv717558130=25, srv63231189=22, srv1409701346=1, srv1726647849=9, srv714706804=24, srv1112494275=0, srv1425259299=2, srv2130286189=13, srv1908517245=11, srv147417365=3, srv2124294864=12, srv1609569731=4, srv559285325=20, srv96596303=28, srv304045273=15, srv984408987=29, srv657946362=23, srv351557284=17, srv870615803=27, srv171616298=8, srv543076232=19, srv1687121652=6, srv566777636=21, srv1883439434=10, srv819493842=26, srv432507554=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:53:17,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:53:17,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:53:17,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-04 02:53:17,677 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0637676837209442, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:53:17,677 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912393973153479, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0637676837209442, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-04 02:55:17,658 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5887031 different iterations. Found a solution that moves 1845 regions; Going from a computed imbalance of 0.9912393973153479 to a new imbalance of 4.306322040866546E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:55:17,666 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1112494275:90 , srv1409701346:90 , srv1425259299:90 , srv147417365:90 , srv1609569731:90 , srv1645578510:90 , srv1687121652:90 , srv1692649306:90 , srv171616298:90 , srv1726647849:90 , srv1883439434:90 , srv1908517245:90 , srv2124294864:90 , srv2130286189:90 , srv262375415:90 , srv304045273:90 , srv313738644:90 , srv351557284:90 , srv432507554:90 , srv543076232:90 , srv559285325:90 , srv566777636:90 , srv63231189:90 , srv657946362:90 , srv714706804:90 , srv717558130:90 , srv819493842:90 , srv870615803:90 , srv96596303:90 , srv984408987:90 } 2023-09-04 02:55:17,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-04 02:55:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1645578510=5, srv1692649306=7, srv262375415=14, srv313738644=16, srv717558130=25, srv63231189=22, srv1409701346=1, srv1726647849=9, srv714706804=24, srv1112494275=0, srv1425259299=2, srv2130286189=13, srv1908517245=11, srv147417365=3, srv2124294864=12, srv1609569731=4, srv559285325=20, srv96596303=28, srv304045273=15, srv984408987=29, srv657946362=23, srv351557284=17, srv870615803=27, srv171616298=8, srv543076232=19, srv1687121652=6, srv566777636=21, srv1883439434=10, srv819493842=26, srv432507554=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-04 02:55:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-04 02:55:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-04 02:55:17,691 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-04 02:55:17,692 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1112494275:90 , srv1409701346:90 , srv1425259299:90 , srv147417365:90 , srv1609569731:90 , srv1645578510:90 , srv1687121652:90 , srv1692649306:90 , srv171616298:90 , srv1726647849:90 , srv1883439434:90 , srv1908517245:90 , srv2124294864:90 , srv2130286189:90 , srv262375415:90 , srv304045273:90 , srv313738644:90 , srv351557284:90 , srv432507554:90 , srv543076232:90 , srv559285325:90 , srv566777636:90 , srv63231189:90 , srv657946362:90 , srv714706804:90 , srv717558130:90 , srv819493842:90 , srv870615803:90 , srv96596303:90 , srv984408987:90 } 2023-09-04 02:55:17,712 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=183 (was 361), ProcessCount=167 (was 169), AvailableMemoryMB=9494 (was 9346) - AvailableMemoryMB LEAK? -