2023-08-05 03:53:16,225 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-05 03:53:16,533 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-05 03:53:16,650 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-05 03:53:16,725 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-05 03:53:16,735 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-05 03:53:16,751 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=171, AvailableMemoryMB=9597 2023-08-05 03:53:16,753 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-05 03:53:16,754 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-05 03:53:16,897 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001920555:84 , srv102708470:84 , srv1036427531:84 , srv1037391483:84 , srv1044215125:84 , srv1052547398:84 , srv1066428842:84 , srv1088973357:84 , srv1113370622:84 , srv1123597270:84 , srv1126893747:84 , srv1129310883:84 , srv1146441110:84 , srv1152776063:84 , srv1169008637:84 , srv1174926615:84 , srv1199552460:84 , srv1202431944:84 , srv1207123333:84 , srv1254121265:84 , srv125522417:84 , srv1256243848:84 , srv1256944812:84 , srv1283555504:84 , srv1283705179:84 , srv1285934926:84 , srv1290960814:84 , srv131183532:84 , srv1324968999:84 , srv1361499008:84 , srv1368519173:84 , srv1407969117:84 , srv144649982:84 , srv1450482539:84 , srv1512676464:84 , srv1520286378:84 , srv1524810158:84 , srv1552003924:84 , srv1558017752:84 , srv1577309377:84 , srv15857268:84 , srv1609534040:84 , srv1633476241:84 , srv1645596604:84 , srv1675419177:84 , srv1705717357:84 , srv1732972537:84 , srv1751637219:84 , srv1760535422:84 , srv1762687208:84 , srv1763935054:84 , srv1793012500:84 , srv183517752:84 , srv1863459225:84 , srv1886785796:84 , srv18876245:84 , srv1914854214:84 , srv1928023813:84 , srv1945528202:84 , srv1951453452:84 , srv1965636707:84 , srv2081910996:84 , srv2082713625:84 , srv2090142187:84 , srv21773901:84 , srv252673566:84 , srv273537395:84 , srv274375856:84 , srv307992340:84 , srv410824082:84 , srv423995560:84 , srv450429302:84 , srv457171242:84 , srv533074761:84 , srv576286400:84 , srv586442929:84 , srv606665917:84 , srv611479180:84 , srv613042832:84 , srv639590896:84 , srv662970897:84 , srv680793578:84 , srv685332306:84 , srv728367740:84 , srv729167085:84 , srv765605948:84 , srv768935644:84 , srv775000006:84 , srv798408521:84 , srv82282466:84 , srv899010591:84 , srv916263982:84 , srv930457828:84 , srv953793496:84 , srv953819836:84 , srv96037800:84 , srv966633651:84 , srv966869614:84 , srv972083673:84 , srv1772796482:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-05 03:53:16,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:16,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2090142187=64, srv685332306=83, srv1368519173=30, srv274375856=68, srv1512676464=34, srv1762687208=49, srv1052547398=5, srv899010591=91, srv1152776063=13, srv183517752=53, srv1202431944=17, srv930457828=93, srv15857268=40, srv1129310883=11, srv613042832=79, srv1450482539=33, srv1793012500=52, srv953793496=94, srv1088973357=7, srv1283555504=23, srv1174926615=15, srv1001920555=0, srv1886785796=55, srv729167085=85, srv1763935054=50, srv2081910996=62, srv144649982=32, srv1965636707=61, srv1524810158=36, srv1169008637=14, srv1256944812=22, srv450429302=72, srv1705717357=45, srv576286400=75, srv798408521=89, srv1254121265=19, srv1123597270=9, srv1256243848=21, srv916263982=92, srv775000006=88, srv768935644=87, srv252673566=66, srv972083673=99, srv1283705179=24, srv1645596604=43, srv1207123333=18, srv1760535422=48, srv639590896=80, srv410824082=70, srv1772796482=51, srv728367740=84, srv1113370622=8, srv966633651=97, srv1290960814=26, srv1609534040=41, srv82282466=90, srv1633476241=42, srv966869614=98, srv1044215125=4, srv1751637219=47, srv1146441110=12, srv1951453452=60, srv1407969117=31, srv125522417=20, srv1361499008=29, srv611479180=78, srv423995560=71, srv1037391483=3, srv1928023813=58, srv102708470=1, srv1577309377=39, srv131183532=27, srv1324968999=28, srv1552003924=37, srv18876245=56, srv21773901=65, srv765605948=86, srv606665917=77, srv2082713625=63, srv586442929=76, srv680793578=82, srv307992340=69, srv662970897=81, srv1945528202=59, srv1675419177=44, srv1732972537=46, srv1126893747=10, srv1285934926=25, srv533074761=74, srv953819836=95, srv1199552460=16, srv457171242=73, srv1558017752=38, srv96037800=96, srv273537395=67, srv1036427531=2, srv1914854214=57, srv1066428842=6, srv1520286378=35, srv1863459225=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 03:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 03:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 03:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 03:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 03:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 03:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 03:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 03:53:17,172 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:17,172 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 03:53:17,172 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 03:53:17,545 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 516 ms to try 5000 different iterations. Found a solution that moves 4660 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18351381957844629. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038415427960322594, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05997567782372504, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038415427960322594, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.243, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:17,556 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv450429302:82 , srv1036427531:83 , srv125522417:83 , srv1256944812:83 , srv1290960814:83 , srv1645596604:83 , srv2090142187:83 , srv252673566:83 , srv423995560:83 , srv586442929:83 , srv611479180:83 , srv639590896:83 , srv680793578:83 , srv1001920555:84 , srv1044215125:84 , srv1129310883:84 , srv1202431944:84 , srv1207123333:84 , srv1283555504:84 , srv1763935054:84 , srv1793012500:84 , srv1951453452:84 , srv533074761:84 , srv775000006:84 , srv102708470:85 , srv1152776063:85 , srv1174926615:85 , srv1199552460:85 , srv1283705179:85 , srv1512676464:85 , srv1520286378:85 , srv1558017752:85 , srv1577309377:85 , srv1633476241:85 , srv1675419177:85 , srv183517752:85 , srv1928023813:85 , srv2081910996:85 , srv899010591:85 , srv953793496:85 , srv966633651:85 , srv1066428842:86 , srv1088973357:86 , srv1126893747:86 , srv131183532:86 , srv1324968999:86 , srv144649982:86 , srv1524810158:86 , srv1751637219:86 , srv1863459225:86 , srv21773901:86 , srv273537395:86 , srv576286400:86 , srv798408521:86 , srv96037800:86 , srv972083673:86 , srv1113370622:87 , srv1123597270:87 , srv1169008637:87 , srv1368519173:87 , srv1450482539:87 , srv1732972537:87 , srv1762687208:87 , srv2082713625:87 , srv274375856:87 , srv307992340:87 , srv662970897:87 , srv685332306:87 , srv729167085:87 , srv916263982:87 , srv1052547398:88 , srv1254121265:88 , srv1361499008:88 , srv1609534040:88 , srv410824082:88 , srv457171242:88 , srv606665917:88 , srv728367740:88 , srv768935644:88 , srv82282466:88 , srv1146441110:89 , srv1285934926:89 , srv1705717357:89 , srv18876245:89 , srv966869614:89 , srv1037391483:90 , srv1256243848:90 , srv1407969117:90 , srv1760535422:90 , srv1945528202:90 , srv953819836:90 , srv1552003924:91 , srv1914854214:91 , srv765605948:91 , srv1965636707:92 , srv15857268:94 , srv930457828:94 , srv1886785796:96 , srv613042832:100 , srv1772796482:431 } 2023-08-05 03:53:17,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:17,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2090142187=64, srv685332306=83, srv1368519173=30, srv274375856=68, srv1512676464=34, srv1762687208=49, srv1052547398=5, srv899010591=91, srv1152776063=13, srv183517752=53, srv1202431944=17, srv930457828=93, srv15857268=40, srv1129310883=11, srv613042832=79, srv1450482539=33, srv1793012500=52, srv953793496=94, srv1088973357=7, srv1283555504=23, srv1174926615=15, srv1001920555=0, srv1886785796=55, srv729167085=85, srv1763935054=50, srv2081910996=62, srv144649982=32, srv1965636707=61, srv1524810158=36, srv1169008637=14, srv1256944812=22, srv450429302=72, srv1705717357=45, srv576286400=75, srv798408521=89, srv1254121265=19, srv1123597270=9, srv1256243848=21, srv916263982=92, srv775000006=88, srv768935644=87, srv252673566=66, srv972083673=99, srv1283705179=24, srv1645596604=43, srv1207123333=18, srv1760535422=48, srv639590896=80, srv410824082=70, srv1772796482=51, srv728367740=84, srv1113370622=8, srv966633651=97, srv1290960814=26, srv1609534040=41, srv82282466=90, srv1633476241=42, srv966869614=98, srv1044215125=4, srv1751637219=47, srv1146441110=12, srv1951453452=60, srv1407969117=31, srv125522417=20, srv1361499008=29, srv611479180=78, srv423995560=71, srv1037391483=3, srv1928023813=58, srv102708470=1, srv1577309377=39, srv131183532=27, srv1324968999=28, srv1552003924=37, srv18876245=56, srv21773901=65, srv765605948=86, srv606665917=77, srv2082713625=63, srv586442929=76, srv680793578=82, srv307992340=69, srv662970897=81, srv1945528202=59, srv1675419177=44, srv1732972537=46, srv1126893747=10, srv1285934926=25, srv533074761=74, srv953819836=95, srv1199552460=16, srv457171242=73, srv1558017752=38, srv96037800=96, srv273537395=67, srv1036427531=2, srv1914854214=57, srv1066428842=6, srv1520286378=35, srv1863459225=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 03:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 03:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 03:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 03:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 03:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 03:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 03:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 03:53:17,737 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038415427960322594, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05997567782372504, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038415427960322594, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.243, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:17,737 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 03:53:17,737 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18348118956103096, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038415427960322594, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05997567782372504, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038415427960322594, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.243, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 03:53:18,024 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 408 ms to try 5000 different iterations. Found a solution that moves 2642 regions; Going from a computed imbalance of 0.18348118956103096 to a new imbalance of 0.014230440773123867. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021819013161199444); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03889321765229827, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29355555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021819013161199444); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:18,034 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv613042832:74 , srv1886785796:77 , srv930457828:77 , srv15857268:78 , srv1552003924:79 , srv1914854214:80 , srv953819836:80 , srv1760535422:81 , srv765605948:81 , srv1407969117:82 , srv1705717357:82 , srv1945528202:82 , srv1965636707:82 , srv1037391483:83 , srv1256243848:83 , srv1361499008:83 , srv1762687208:83 , srv274375856:83 , srv82282466:83 , srv966869614:83 , srv1066428842:84 , srv1146441110:84 , srv1169008637:84 , srv1254121265:84 , srv1285934926:84 , srv18876245:84 , srv410824082:84 , srv728367740:84 , srv102708470:85 , srv1052547398:85 , srv1088973357:85 , srv1123597270:85 , srv131183532:85 , srv1324968999:85 , srv1520286378:85 , srv1524810158:85 , srv1863459225:85 , srv21773901:85 , srv273537395:85 , srv606665917:85 , srv685332306:85 , srv768935644:85 , srv972083673:85 , srv1001920555:86 , srv1113370622:86 , srv1207123333:86 , srv1283705179:86 , srv1368519173:86 , srv1450482539:86 , srv1558017752:86 , srv1609534040:86 , srv1633476241:86 , srv1732972537:86 , srv183517752:86 , srv2081910996:86 , srv457171242:86 , srv576286400:86 , srv662970897:86 , srv729167085:86 , srv798408521:86 , srv916263982:86 , srv1126893747:87 , srv1152776063:87 , srv1290960814:87 , srv1763935054:87 , srv1928023813:87 , srv1951453452:87 , srv2082713625:87 , srv307992340:87 , srv423995560:87 , srv775000006:87 , srv953793496:87 , srv966633651:87 , srv1044215125:88 , srv1174926615:88 , srv1199552460:88 , srv144649982:88 , srv1577309377:88 , srv1675419177:88 , srv1751637219:88 , srv1793012500:88 , srv2090142187:88 , srv586442929:88 , srv680793578:88 , srv1036427531:89 , srv1129310883:89 , srv1202431944:89 , srv125522417:89 , srv1256944812:89 , srv1645596604:89 , srv252673566:89 , srv450429302:89 , srv639590896:89 , srv899010591:89 , srv96037800:89 , srv1283555504:90 , srv1512676464:90 , srv533074761:90 , srv611479180:91 , srv1772796482:537 } 2023-08-05 03:53:18,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:18,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2090142187=64, srv685332306=83, srv1368519173=30, srv274375856=68, srv1512676464=34, srv1762687208=49, srv1052547398=5, srv899010591=91, srv1152776063=13, srv183517752=53, srv1202431944=17, srv930457828=93, srv15857268=40, srv1129310883=11, srv613042832=79, srv1450482539=33, srv1793012500=52, srv953793496=94, srv1088973357=7, srv1283555504=23, srv1174926615=15, srv1001920555=0, srv1886785796=55, srv729167085=85, srv1763935054=50, srv2081910996=62, srv144649982=32, srv1965636707=61, srv1524810158=36, srv1169008637=14, srv1256944812=22, srv450429302=72, srv1705717357=45, srv576286400=75, srv798408521=89, srv1254121265=19, srv1123597270=9, srv1256243848=21, srv916263982=92, srv775000006=88, srv768935644=87, srv252673566=66, srv972083673=99, srv1283705179=24, srv1645596604=43, srv1207123333=18, srv1760535422=48, srv639590896=80, srv410824082=70, srv1772796482=51, srv728367740=84, srv1113370622=8, srv966633651=97, srv1290960814=26, srv1609534040=41, srv82282466=90, srv1633476241=42, srv966869614=98, srv1044215125=4, srv1751637219=47, srv1146441110=12, srv1951453452=60, srv1407969117=31, srv125522417=20, srv1361499008=29, srv611479180=78, srv423995560=71, srv1037391483=3, srv1928023813=58, srv102708470=1, srv1577309377=39, srv131183532=27, srv1324968999=28, srv1552003924=37, srv18876245=56, srv21773901=65, srv765605948=86, srv606665917=77, srv2082713625=63, srv586442929=76, srv680793578=82, srv307992340=69, srv662970897=81, srv1945528202=59, srv1675419177=44, srv1732972537=46, srv1126893747=10, srv1285934926=25, srv533074761=74, srv953819836=95, srv1199552460=16, srv457171242=73, srv1558017752=38, srv96037800=96, srv273537395=67, srv1036427531=2, srv1914854214=57, srv1066428842=6, srv1520286378=35, srv1863459225=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:18,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:18,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:18,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 03:53:18,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 03:53:18,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 03:53:18,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 03:53:18,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 03:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 03:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 03:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 03:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 03:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 03:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 03:53:18,181 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021819013161199444); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03889321765229827, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021819013161199444); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:18,181 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 03:53:18,181 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014211941093722293, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021819013161199444); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03889321765229827, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021819013161199444); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 03:53:18,526 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 450 ms to try 5000 different iterations. Found a solution that moves 1011 regions; Going from a computed imbalance of 0.014211941093722293 to a new imbalance of 7.354312585672784E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006661052630583241); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008012336167697753); MoveCostFunction : (multiplier=7.0, imbalance=0.11233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006661052630583241); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:18,528 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1512676464:82 , srv1609534040:82 , srv1965636707:82 , srv1283555504:83 , srv1751637219:83 , srv307992340:83 , srv457171242:83 , srv899010591:83 , srv96037800:83 , srv1037391483:84 , srv1052547398:84 , srv1126893747:84 , srv1129310883:84 , srv1174926615:84 , srv1202431944:84 , srv144649982:84 , srv1577309377:84 , srv15857268:84 , srv1732972537:84 , srv2082713625:84 , srv533074761:84 , srv611479180:84 , srv613042832:84 , srv685332306:84 , srv1036427531:85 , srv1044215125:85 , srv1113370622:85 , srv1152776063:85 , srv1199552460:85 , srv1254121265:85 , srv1256243848:85 , srv1256944812:85 , srv1285934926:85 , srv1368519173:85 , srv1450482539:85 , srv1675419177:85 , srv1793012500:85 , srv18876245:85 , srv1914854214:85 , srv1928023813:85 , srv1945528202:85 , srv252673566:85 , srv606665917:85 , srv639590896:85 , srv662970897:85 , srv729167085:85 , srv768935644:85 , srv798408521:85 , srv916263982:85 , srv953793496:85 , srv1088973357:86 , srv1123597270:86 , srv1146441110:86 , srv1169008637:86 , srv125522417:86 , srv1283705179:86 , srv1290960814:86 , srv131183532:86 , srv1324968999:86 , srv1361499008:86 , srv1407969117:86 , srv1645596604:86 , srv1763935054:86 , srv183517752:86 , srv1863459225:86 , srv1886785796:86 , srv1951453452:86 , srv2081910996:86 , srv2090142187:86 , srv273537395:86 , srv410824082:86 , srv450429302:86 , srv576286400:86 , srv586442929:86 , srv680793578:86 , srv728367740:86 , srv765605948:86 , srv775000006:86 , srv82282466:86 , srv966633651:86 , srv966869614:86 , srv1066428842:87 , srv1207123333:87 , srv1520286378:87 , srv1552003924:87 , srv1558017752:87 , srv1633476241:87 , srv1705717357:87 , srv1760535422:87 , srv1762687208:87 , srv21773901:87 , srv423995560:87 , srv930457828:87 , srv972083673:87 , srv1001920555:88 , srv102708470:88 , srv1524810158:88 , srv274375856:88 , srv953819836:88 , srv1772796482:549 } 2023-08-05 03:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2090142187=64, srv685332306=83, srv1368519173=30, srv274375856=68, srv1512676464=34, srv1762687208=49, srv1052547398=5, srv899010591=91, srv1152776063=13, srv183517752=53, srv1202431944=17, srv930457828=93, srv15857268=40, srv1129310883=11, srv613042832=79, srv1450482539=33, srv1793012500=52, srv953793496=94, srv1088973357=7, srv1283555504=23, srv1174926615=15, srv1001920555=0, srv1886785796=55, srv729167085=85, srv1763935054=50, srv2081910996=62, srv144649982=32, srv1965636707=61, srv1524810158=36, srv1169008637=14, srv1256944812=22, srv450429302=72, srv1705717357=45, srv576286400=75, srv798408521=89, srv1254121265=19, srv1123597270=9, srv1256243848=21, srv916263982=92, srv775000006=88, srv768935644=87, srv252673566=66, srv972083673=99, srv1283705179=24, srv1645596604=43, srv1207123333=18, srv1760535422=48, srv639590896=80, srv410824082=70, srv1772796482=51, srv728367740=84, srv1113370622=8, srv966633651=97, srv1290960814=26, srv1609534040=41, srv82282466=90, srv1633476241=42, srv966869614=98, srv1044215125=4, srv1751637219=47, srv1146441110=12, srv1951453452=60, srv1407969117=31, srv125522417=20, srv1361499008=29, srv611479180=78, srv423995560=71, srv1037391483=3, srv1928023813=58, srv102708470=1, srv1577309377=39, srv131183532=27, srv1324968999=28, srv1552003924=37, srv18876245=56, srv21773901=65, srv765605948=86, srv606665917=77, srv2082713625=63, srv586442929=76, srv680793578=82, srv307992340=69, srv662970897=81, srv1945528202=59, srv1675419177=44, srv1732972537=46, srv1126893747=10, srv1285934926=25, srv533074761=74, srv953819836=95, srv1199552460=16, srv457171242=73, srv1558017752=38, srv96037800=96, srv273537395=67, srv1036427531=2, srv1914854214=57, srv1066428842=6, srv1520286378=35, srv1863459225=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:18,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 03:53:18,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 03:53:18,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 03:53:18,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 03:53:18,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 03:53:18,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 03:53:18,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 03:53:18,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 03:53:18,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 03:53:18,680 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006661052630583241); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008012336167697753); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006661052630583241); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:18,680 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 03:53:18,680 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.283520852610734E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006661052630583241); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008012336167697753); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006661052630583241); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 03:53:18,984 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 406 ms to try 5000 different iterations. Found a solution that moves 296 regions; Going from a computed imbalance of 7.283520852610734E-4 to a new imbalance of 3.588809275083706E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.03288888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:18,986 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv102708470:83 , srv1524810158:83 , srv1886785796:83 , srv576286400:83 , srv1001920555:84 , srv1113370622:84 , srv1123597270:84 , srv1146441110:84 , srv1199552460:84 , srv1254121265:84 , srv125522417:84 , srv1256243848:84 , srv1285934926:84 , srv1368519173:84 , srv1407969117:84 , srv144649982:84 , srv1450482539:84 , srv1558017752:84 , srv1633476241:84 , srv1645596604:84 , srv1675419177:84 , srv1705717357:84 , srv1760535422:84 , srv18876245:84 , srv1945528202:84 , srv2082713625:84 , srv273537395:84 , srv274375856:84 , srv533074761:84 , srv606665917:84 , srv611479180:84 , srv613042832:84 , srv662970897:84 , srv728367740:84 , srv729167085:84 , srv765605948:84 , srv768935644:84 , srv775000006:84 , srv916263982:84 , srv930457828:84 , srv953819836:84 , srv96037800:84 , srv966633651:84 , srv966869614:84 , srv972083673:84 , srv1036427531:85 , srv1037391483:85 , srv1044215125:85 , srv1066428842:85 , srv1088973357:85 , srv1126893747:85 , srv1129310883:85 , srv1152776063:85 , srv1169008637:85 , srv1174926615:85 , srv1202431944:85 , srv1207123333:85 , srv1283555504:85 , srv1283705179:85 , srv131183532:85 , srv1324968999:85 , srv1361499008:85 , srv1520286378:85 , srv1552003924:85 , srv1577309377:85 , srv1732972537:85 , srv1751637219:85 , srv1762687208:85 , srv1763935054:85 , srv1793012500:85 , srv183517752:85 , srv1863459225:85 , srv1928023813:85 , srv1951453452:85 , srv2081910996:85 , srv2090142187:85 , srv21773901:85 , srv307992340:85 , srv410824082:85 , srv423995560:85 , srv450429302:85 , srv457171242:85 , srv586442929:85 , srv639590896:85 , srv680793578:85 , srv685332306:85 , srv798408521:85 , srv82282466:85 , srv899010591:85 , srv953793496:85 , srv1052547398:86 , srv1256944812:86 , srv1290960814:86 , srv1512676464:86 , srv15857268:86 , srv1609534040:86 , srv1914854214:86 , srv1965636707:86 , srv252673566:86 , srv1772796482:625 } 2023-08-05 03:53:18,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:18,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2090142187=64, srv685332306=83, srv1368519173=30, srv274375856=68, srv1512676464=34, srv1762687208=49, srv1052547398=5, srv899010591=91, srv1152776063=13, srv183517752=53, srv1202431944=17, srv930457828=93, srv15857268=40, srv1129310883=11, srv613042832=79, srv1450482539=33, srv1793012500=52, srv953793496=94, srv1088973357=7, srv1283555504=23, srv1174926615=15, srv1001920555=0, srv1886785796=55, srv729167085=85, srv1763935054=50, srv2081910996=62, srv144649982=32, srv1965636707=61, srv1524810158=36, srv1169008637=14, srv1256944812=22, srv450429302=72, srv1705717357=45, srv576286400=75, srv798408521=89, srv1254121265=19, srv1123597270=9, srv1256243848=21, srv916263982=92, srv775000006=88, srv768935644=87, srv252673566=66, srv972083673=99, srv1283705179=24, srv1645596604=43, srv1207123333=18, srv1760535422=48, srv639590896=80, srv410824082=70, srv1772796482=51, srv728367740=84, srv1113370622=8, srv966633651=97, srv1290960814=26, srv1609534040=41, srv82282466=90, srv1633476241=42, srv966869614=98, srv1044215125=4, srv1751637219=47, srv1146441110=12, srv1951453452=60, srv1407969117=31, srv125522417=20, srv1361499008=29, srv611479180=78, srv423995560=71, srv1037391483=3, srv1928023813=58, srv102708470=1, srv1577309377=39, srv131183532=27, srv1324968999=28, srv1552003924=37, srv18876245=56, srv21773901=65, srv765605948=86, srv606665917=77, srv2082713625=63, srv586442929=76, srv680793578=82, srv307992340=69, srv662970897=81, srv1945528202=59, srv1675419177=44, srv1732972537=46, srv1126893747=10, srv1285934926=25, srv533074761=74, srv953819836=95, srv1199552460=16, srv457171242=73, srv1558017752=38, srv96037800=96, srv273537395=67, srv1036427531=2, srv1914854214=57, srv1066428842=6, srv1520286378=35, srv1863459225=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 03:53:19,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 03:53:19,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 03:53:19,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 03:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 03:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 03:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 03:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 03:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 03:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 03:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 03:53:19,127 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:19,127 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 03:53:19,128 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.381545645149316E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 03:53:19,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 451 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 3.381545645149316E-5 to a new imbalance of 7.583817690791035E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:19,476 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1052547398:83 , srv1146441110:83 , srv1256944812:83 , srv1512676464:83 , srv21773901:83 , srv252673566:83 , srv410824082:83 , srv1001920555:84 , srv1036427531:84 , srv1037391483:84 , srv1044215125:84 , srv1066428842:84 , srv1088973357:84 , srv1113370622:84 , srv1123597270:84 , srv1126893747:84 , srv1129310883:84 , srv1152776063:84 , srv1169008637:84 , srv1174926615:84 , srv1199552460:84 , srv1202431944:84 , srv1207123333:84 , srv125522417:84 , srv1256243848:84 , srv1283555504:84 , srv1283705179:84 , srv1285934926:84 , srv1290960814:84 , srv131183532:84 , srv1324968999:84 , srv1361499008:84 , srv1368519173:84 , srv1407969117:84 , srv144649982:84 , srv1450482539:84 , srv1520286378:84 , srv1524810158:84 , srv1552003924:84 , srv1558017752:84 , srv1577309377:84 , srv15857268:84 , srv1609534040:84 , srv1633476241:84 , srv1645596604:84 , srv1675419177:84 , srv1705717357:84 , srv1732972537:84 , srv1751637219:84 , srv1760535422:84 , srv1762687208:84 , srv1763935054:84 , srv1793012500:84 , srv183517752:84 , srv1863459225:84 , srv1886785796:84 , srv18876245:84 , srv1914854214:84 , srv1928023813:84 , srv1951453452:84 , srv1965636707:84 , srv2081910996:84 , srv2082713625:84 , srv2090142187:84 , srv274375856:84 , srv307992340:84 , srv423995560:84 , srv450429302:84 , srv457171242:84 , srv533074761:84 , srv586442929:84 , srv606665917:84 , srv611479180:84 , srv613042832:84 , srv639590896:84 , srv662970897:84 , srv680793578:84 , srv728367740:84 , srv729167085:84 , srv765605948:84 , srv768935644:84 , srv798408521:84 , srv82282466:84 , srv899010591:84 , srv916263982:84 , srv930457828:84 , srv953793496:84 , srv953819836:84 , srv96037800:84 , srv966633651:84 , srv966869614:84 , srv972083673:84 , srv102708470:85 , srv1254121265:85 , srv1945528202:85 , srv273537395:85 , srv576286400:85 , srv685332306:85 , srv775000006:85 , srv1772796482:684 } 2023-08-05 03:53:19,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:19,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2090142187=64, srv685332306=83, srv1368519173=30, srv274375856=68, srv1512676464=34, srv1762687208=49, srv1052547398=5, srv899010591=91, srv1152776063=13, srv183517752=53, srv1202431944=17, srv930457828=93, srv15857268=40, srv1129310883=11, srv613042832=79, srv1450482539=33, srv1793012500=52, srv953793496=94, srv1088973357=7, srv1283555504=23, srv1174926615=15, srv1001920555=0, srv1886785796=55, srv729167085=85, srv1763935054=50, srv2081910996=62, srv144649982=32, srv1965636707=61, srv1524810158=36, srv1169008637=14, srv1256944812=22, srv450429302=72, srv1705717357=45, srv576286400=75, srv798408521=89, srv1254121265=19, srv1123597270=9, srv1256243848=21, srv916263982=92, srv775000006=88, srv768935644=87, srv252673566=66, srv972083673=99, srv1283705179=24, srv1645596604=43, srv1207123333=18, srv1760535422=48, srv639590896=80, srv410824082=70, srv1772796482=51, srv728367740=84, srv1113370622=8, srv966633651=97, srv1290960814=26, srv1609534040=41, srv82282466=90, srv1633476241=42, srv966869614=98, srv1044215125=4, srv1751637219=47, srv1146441110=12, srv1951453452=60, srv1407969117=31, srv125522417=20, srv1361499008=29, srv611479180=78, srv423995560=71, srv1037391483=3, srv1928023813=58, srv102708470=1, srv1577309377=39, srv131183532=27, srv1324968999=28, srv1552003924=37, srv18876245=56, srv21773901=65, srv765605948=86, srv606665917=77, srv2082713625=63, srv586442929=76, srv680793578=82, srv307992340=69, srv662970897=81, srv1945528202=59, srv1675419177=44, srv1732972537=46, srv1126893747=10, srv1285934926=25, srv533074761=74, srv953819836=95, srv1199552460=16, srv457171242=73, srv1558017752=38, srv96037800=96, srv273537395=67, srv1036427531=2, srv1914854214=57, srv1066428842=6, srv1520286378=35, srv1863459225=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 03:53:19,606 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:19,606 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1052547398:83 , srv1146441110:83 , srv1256944812:83 , srv1512676464:83 , srv21773901:83 , srv252673566:83 , srv410824082:83 , srv1001920555:84 , srv1036427531:84 , srv1037391483:84 , srv1044215125:84 , srv1066428842:84 , srv1088973357:84 , srv1113370622:84 , srv1123597270:84 , srv1126893747:84 , srv1129310883:84 , srv1152776063:84 , srv1169008637:84 , srv1174926615:84 , srv1199552460:84 , srv1202431944:84 , srv1207123333:84 , srv125522417:84 , srv1256243848:84 , srv1283555504:84 , srv1283705179:84 , srv1285934926:84 , srv1290960814:84 , srv131183532:84 , srv1324968999:84 , srv1361499008:84 , srv1368519173:84 , srv1407969117:84 , srv144649982:84 , srv1450482539:84 , srv1520286378:84 , srv1524810158:84 , srv1552003924:84 , srv1558017752:84 , srv1577309377:84 , srv15857268:84 , srv1609534040:84 , srv1633476241:84 , srv1645596604:84 , srv1675419177:84 , srv1705717357:84 , srv1732972537:84 , srv1751637219:84 , srv1760535422:84 , srv1762687208:84 , srv1763935054:84 , srv1793012500:84 , srv183517752:84 , srv1863459225:84 , srv1886785796:84 , srv18876245:84 , srv1914854214:84 , srv1928023813:84 , srv1951453452:84 , srv1965636707:84 , srv2081910996:84 , srv2082713625:84 , srv2090142187:84 , srv274375856:84 , srv307992340:84 , srv423995560:84 , srv450429302:84 , srv457171242:84 , srv533074761:84 , srv586442929:84 , srv606665917:84 , srv611479180:84 , srv613042832:84 , srv639590896:84 , srv662970897:84 , srv680793578:84 , srv728367740:84 , srv729167085:84 , srv765605948:84 , srv768935644:84 , srv798408521:84 , srv82282466:84 , srv899010591:84 , srv916263982:84 , srv930457828:84 , srv953793496:84 , srv953819836:84 , srv96037800:84 , srv966633651:84 , srv966869614:84 , srv972083673:84 , srv102708470:85 , srv1254121265:85 , srv1945528202:85 , srv273537395:85 , srv576286400:85 , srv685332306:85 , srv775000006:85 , srv1772796482:684 } 2023-08-05 03:53:19,636 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=272 (was 272), ProcessCount=171 (was 171), AvailableMemoryMB=8784 (was 9597) 2023-08-05 03:53:19,645 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=171, AvailableMemoryMB=8783 2023-08-05 03:53:19,646 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-05 03:53:19,646 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-05 03:53:19,659 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1118371047:84 , srv1277554375:84 , srv1282160455:84 , srv1299845835:84 , srv135465918:84 , srv1447779429:84 , srv1586633288:84 , srv1598264004:84 , srv1687163890:84 , srv17052958:84 , srv1796585928:84 , srv1837499501:84 , srv185375760:84 , srv1931094460:84 , srv2095061680:84 , srv290829830:84 , srv351861438:84 , srv383787099:84 , srv477994488:84 , srv478838717:84 , srv520820390:84 , srv599964254:84 , srv719462120:84 , srv76432695:84 , srv783115834:84 , srv847893242:84 , srv868882445:84 , srv890222096:84 , srv904302154:84 , srv1325229670:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-05 03:53:19,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:53:19,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1299845835=3, srv1598264004=8, srv783115834=25, srv478838717=20, srv520820390=21, srv351861438=17, srv1586633288=7, srv1837499501=12, srv1447779429=6, srv904302154=29, srv719462120=23, srv599964254=22, srv2095061680=15, srv1118371047=0, srv477994488=19, srv135465918=5, srv868882445=27, srv1277554375=1, srv1282160455=2, srv1796585928=11, srv1325229670=4, srv847893242=26, srv76432695=24, srv1687163890=9, srv17052958=10, srv383787099=18, srv185375760=13, srv890222096=28, srv290829830=16, srv1931094460=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-05 03:53:19,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.140361781801059, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:53:19,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.99126353186555, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.140361781801059, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-05 03:55:19,682 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5464363 different iterations. Found a solution that moves 1865 regions; Going from a computed imbalance of 0.99126353186555 to a new imbalance of 4.353003038599517E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6907407407407408, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:55:19,688 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1118371047:90 , srv1277554375:90 , srv1282160455:90 , srv1299845835:90 , srv1325229670:90 , srv135465918:90 , srv1447779429:90 , srv1586633288:90 , srv1598264004:90 , srv1687163890:90 , srv17052958:90 , srv1796585928:90 , srv1837499501:90 , srv185375760:90 , srv1931094460:90 , srv2095061680:90 , srv290829830:90 , srv351861438:90 , srv383787099:90 , srv477994488:90 , srv478838717:90 , srv520820390:90 , srv599964254:90 , srv719462120:90 , srv76432695:90 , srv783115834:90 , srv847893242:90 , srv868882445:90 , srv890222096:90 , srv904302154:90 } 2023-08-05 03:55:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 03:55:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1299845835=3, srv1598264004=8, srv783115834=25, srv478838717=20, srv520820390=21, srv351861438=17, srv1586633288=7, srv1837499501=12, srv1447779429=6, srv904302154=29, srv719462120=23, srv599964254=22, srv2095061680=15, srv1118371047=0, srv477994488=19, srv135465918=5, srv868882445=27, srv1277554375=1, srv1282160455=2, srv1796585928=11, srv1325229670=4, srv847893242=26, srv76432695=24, srv1687163890=9, srv17052958=10, srv383787099=18, srv185375760=13, srv890222096=28, srv290829830=16, srv1931094460=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 03:55:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 03:55:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 03:55:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 03:55:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-05 03:55:19,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 03:55:19,708 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1118371047:90 , srv1277554375:90 , srv1282160455:90 , srv1299845835:90 , srv1325229670:90 , srv135465918:90 , srv1447779429:90 , srv1586633288:90 , srv1598264004:90 , srv1687163890:90 , srv17052958:90 , srv1796585928:90 , srv1837499501:90 , srv185375760:90 , srv1931094460:90 , srv2095061680:90 , srv290829830:90 , srv351861438:90 , srv383787099:90 , srv477994488:90 , srv478838717:90 , srv520820390:90 , srv599964254:90 , srv719462120:90 , srv76432695:90 , srv783115834:90 , srv847893242:90 , srv868882445:90 , srv890222096:90 , srv904302154:90 } 2023-08-05 03:55:19,720 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=185 (was 272), ProcessCount=171 (was 171), AvailableMemoryMB=8053 (was 8783)