2023-10-20 10:55:33,061 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-20 10:55:33,349 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-20 10:55:33,460 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-20 10:55:33,525 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-20 10:55:33,531 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-20 10:55:33,546 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=434, ProcessCount=175, AvailableMemoryMB=2652 2023-10-20 10:55:33,548 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-20 10:55:33,548 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-20 10:55:33,682 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012485467:84 , srv1023786415:84 , srv1047725234:84 , srv1053356277:84 , srv1060089958:84 , srv1081426314:84 , srv1083446808:84 , srv1087769207:84 , srv1104759231:84 , srv1116334842:84 , srv1175498467:84 , srv122899424:84 , srv1233204713:84 , srv1250178982:84 , srv1284327298:84 , srv1299478500:84 , srv1305326789:84 , srv1351627939:84 , srv1396240289:84 , srv1435682169:84 , srv1435794015:84 , srv1448493677:84 , srv146664366:84 , srv1474352001:84 , srv1484309726:84 , srv148563052:84 , srv1493437973:84 , srv1494687568:84 , srv1508507522:84 , srv1530888195:84 , srv1545751096:84 , srv1570931305:84 , srv1586577997:84 , srv1629907173:84 , srv1634490451:84 , srv1636816046:84 , srv1646513189:84 , srv1654632977:84 , srv1695055295:84 , srv1701964014:84 , srv1730518036:84 , srv17576380:84 , srv1770016068:84 , srv17715125:84 , srv1801527185:84 , srv1806412777:84 , srv1807335:84 , srv1817248716:84 , srv1853740115:84 , srv1952145003:84 , srv1955781188:84 , srv1964089380:84 , srv1971923111:84 , srv1974758747:84 , srv1986363224:84 , srv2029898661:84 , srv2036170954:84 , srv2058167855:84 , srv2062249717:84 , srv2070113940:84 , srv2125799702:84 , srv268236415:84 , srv275969840:84 , srv296448398:84 , srv316311702:84 , srv318374192:84 , srv388382142:84 , srv422128216:84 , srv444177344:84 , srv45940423:84 , srv484846137:84 , srv495065549:84 , srv557324165:84 , srv565706037:84 , srv58536521:84 , srv605448350:84 , srv621029853:84 , srv630541336:84 , srv64058751:84 , srv642513965:84 , srv693475259:84 , srv705654320:84 , srv728474872:84 , srv754397315:84 , srv788520731:84 , srv803726728:84 , srv808418427:84 , srv833911551:84 , srv858903721:84 , srv876075872:84 , srv878714023:84 , srv882011961:84 , srv894258434:84 , srv904723070:84 , srv910965801:84 , srv926487051:84 , srv940229362:84 , srv944186205:84 , srv99325228:84 , srv942083623:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-20 10:55:33,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:33,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv45940423=69, srv944186205=98, srv318374192=65, srv878714023=90, srv1586577997=32, srv1853740115=48, srv388382142=66, srv1448493677=21, srv1629907173=33, srv1654632977=37, srv296448398=63, srv1636816046=35, srv1104759231=8, srv1701964014=39, srv316311702=64, srv904723070=93, srv1474352001=23, srv1299478500=15, srv1305326789=16, srv728474872=82, srv926487051=95, srv122899424=11, srv1484309726=24, srv1986363224=54, srv1116334842=9, srv754397315=83, srv1964089380=51, srv605448350=75, srv803726728=85, srv146664366=22, srv495065549=71, srv1233204713=12, srv1435794015=20, srv148563052=25, srv422128216=67, srv1060089958=4, srv1570931305=31, srv17715125=43, srv833911551=87, srv1952145003=49, srv2125799702=60, srv1971923111=52, srv1435682169=19, srv1396240289=18, srv1807335=46, srv1806412777=45, srv268236415=61, srv876075872=89, srv693475259=80, srv2062249717=58, srv58536521=74, srv1508507522=28, srv1955781188=50, srv642513965=79, srv1081426314=5, srv2058167855=57, srv484846137=70, srv557324165=72, srv1494687568=27, srv942083623=97, srv2036170954=56, srv2070113940=59, srv1770016068=42, srv1730518036=40, srv1012485467=0, srv882011961=91, srv1047725234=2, srv705654320=81, srv621029853=76, srv1545751096=30, srv17576380=41, srv894258434=92, srv1530888195=29, srv1974758747=53, srv2029898661=55, srv1646513189=36, srv1087769207=7, srv64058751=78, srv444177344=68, srv630541336=77, srv1351627939=17, srv1023786415=1, srv1053356277=3, srv1250178982=13, srv1083446808=6, srv1493437973=26, srv858903721=88, srv275969840=62, srv910965801=94, srv1634490451=34, srv1284327298=14, srv99325228=99, srv940229362=96, srv1817248716=47, srv1801527185=44, srv1695055295=38, srv808418427=86, srv1175498467=10, srv788520731=84, srv565706037=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:33,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:33,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:33,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:33,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 10:55:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 10:55:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 10:55:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 10:55:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 10:55:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 10:55:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 10:55:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 10:55:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 10:55:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 10:55:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 10:55:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 10:55:33,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 10:55:33,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 10:55:33,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 10:55:33,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 10:55:33,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 10:55:33,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 10:55:33,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 10:55:33,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 10:55:33,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 10:55:33,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 10:55:33,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 10:55:33,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 10:55:33,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 10:55:33,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 10:55:33,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 10:55:33,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 10:55:33,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 10:55:33,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 10:55:33,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 10:55:33,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 10:55:33,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 10:55:33,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 10:55:33,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 10:55:33,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 10:55:33,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 10:55:33,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 10:55:33,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 10:55:33,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 10:55:33,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 10:55:34,080 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:34,080 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 10:55:34,081 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 10:55:34,411 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 578 ms to try 5000 different iterations. Found a solution that moves 4734 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17492438087125378. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040642980351493066, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054298919096417396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.526, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040642980351493066, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:34,421 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2036170954:80 , srv1493437973:81 , srv1175498467:82 , srv148563052:82 , srv1530888195:82 , srv1586577997:82 , srv1955781188:82 , srv1971923111:83 , srv1986363224:83 , srv316311702:83 , srv621029853:83 , srv940229362:83 , srv944186205:83 , srv122899424:84 , srv1305326789:84 , srv1695055295:84 , srv1964089380:84 , srv2070113940:84 , srv484846137:84 , srv64058751:84 , srv642513965:84 , srv1012485467:85 , srv1299478500:85 , srv1508507522:85 , srv1654632977:85 , srv1770016068:85 , srv1807335:85 , srv1974758747:85 , srv2029898661:85 , srv2058167855:85 , srv296448398:85 , srv318374192:85 , srv495065549:85 , srv605448350:85 , srv728474872:85 , srv803726728:85 , srv882011961:85 , srv1060089958:86 , srv1083446808:86 , srv1087769207:86 , srv1104759231:86 , srv1116334842:86 , srv1250178982:86 , srv1284327298:86 , srv1570931305:86 , srv1629907173:86 , srv1634490451:86 , srv1730518036:86 , srv1817248716:86 , srv1952145003:86 , srv2125799702:86 , srv275969840:86 , srv388382142:86 , srv557324165:86 , srv58536521:86 , srv693475259:86 , srv833911551:86 , srv99325228:86 , srv1023786415:87 , srv1233204713:87 , srv1351627939:87 , srv1435682169:87 , srv1474352001:87 , srv1484309726:87 , srv1636816046:87 , srv1646513189:87 , srv17576380:87 , srv1801527185:87 , srv1853740115:87 , srv2062249717:87 , srv630541336:87 , srv858903721:87 , srv910965801:87 , srv926487051:87 , srv1047725234:88 , srv1053356277:88 , srv146664366:88 , srv17715125:88 , srv422128216:88 , srv45940423:88 , srv565706037:88 , srv878714023:88 , srv1081426314:89 , srv1448493677:89 , srv1494687568:89 , srv1545751096:89 , srv268236415:89 , srv444177344:89 , srv1396240289:90 , srv754397315:90 , srv808418427:90 , srv894258434:90 , srv904723070:90 , srv705654320:91 , srv788520731:91 , srv1435794015:92 , srv1701964014:92 , srv1806412777:97 , srv876075872:99 , srv942083623:451 } 2023-10-20 10:55:34,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:34,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv45940423=69, srv944186205=98, srv318374192=65, srv878714023=90, srv1586577997=32, srv1853740115=48, srv388382142=66, srv1448493677=21, srv1629907173=33, srv1654632977=37, srv296448398=63, srv1636816046=35, srv1104759231=8, srv1701964014=39, srv316311702=64, srv904723070=93, srv1474352001=23, srv1299478500=15, srv1305326789=16, srv728474872=82, srv926487051=95, srv122899424=11, srv1484309726=24, srv1986363224=54, srv1116334842=9, srv754397315=83, srv1964089380=51, srv605448350=75, srv803726728=85, srv146664366=22, srv495065549=71, srv1233204713=12, srv1435794015=20, srv148563052=25, srv422128216=67, srv1060089958=4, srv1570931305=31, srv17715125=43, srv833911551=87, srv1952145003=49, srv2125799702=60, srv1971923111=52, srv1435682169=19, srv1396240289=18, srv1807335=46, srv1806412777=45, srv268236415=61, srv876075872=89, srv693475259=80, srv2062249717=58, srv58536521=74, srv1508507522=28, srv1955781188=50, srv642513965=79, srv1081426314=5, srv2058167855=57, srv484846137=70, srv557324165=72, srv1494687568=27, srv942083623=97, srv2036170954=56, srv2070113940=59, srv1770016068=42, srv1730518036=40, srv1012485467=0, srv882011961=91, srv1047725234=2, srv705654320=81, srv621029853=76, srv1545751096=30, srv17576380=41, srv894258434=92, srv1530888195=29, srv1974758747=53, srv2029898661=55, srv1646513189=36, srv1087769207=7, srv64058751=78, srv444177344=68, srv630541336=77, srv1351627939=17, srv1023786415=1, srv1053356277=3, srv1250178982=13, srv1083446808=6, srv1493437973=26, srv858903721=88, srv275969840=62, srv910965801=94, srv1634490451=34, srv1284327298=14, srv99325228=99, srv940229362=96, srv1817248716=47, srv1801527185=44, srv1695055295=38, srv808418427=86, srv1175498467=10, srv788520731=84, srv565706037=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:34,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:34,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:34,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:34,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:34,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:34,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:34,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:34,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:34,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 10:55:34,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 10:55:34,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 10:55:34,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 10:55:34,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 10:55:34,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 10:55:34,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 10:55:34,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 10:55:34,611 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040642980351493066, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054298919096417396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040642980351493066, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:34,611 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 10:55:34,611 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17489123269476362, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040642980351493066, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054298919096417396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040642980351493066, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 10:55:34,818 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 337 ms to try 5000 different iterations. Found a solution that moves 2600 regions; Going from a computed imbalance of 0.17489123269476362 to a new imbalance of 0.015295921065235365. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0246355148534342); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039312269655344824, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0246355148534342); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:34,829 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1806412777:74 , srv876075872:74 , srv1701964014:79 , srv444177344:80 , srv1435794015:81 , srv705654320:81 , srv788520731:81 , srv808418427:81 , srv894258434:81 , srv1047725234:82 , srv1396240289:82 , srv754397315:82 , srv1023786415:83 , srv1053356277:83 , srv1081426314:83 , srv1351627939:83 , srv146664366:83 , srv1474352001:83 , srv1494687568:83 , srv17715125:83 , srv565706037:83 , srv858903721:83 , srv904723070:83 , srv926487051:83 , srv1012485467:84 , srv1087769207:84 , srv1104759231:84 , srv1484309726:84 , srv1545751096:84 , srv17576380:84 , srv1853740115:84 , srv2062249717:84 , srv268236415:84 , srv275969840:84 , srv630541336:84 , srv1250178982:85 , srv1508507522:85 , srv1570931305:85 , srv1636816046:85 , srv1646513189:85 , srv1730518036:85 , srv1770016068:85 , srv1801527185:85 , srv2070113940:85 , srv2125799702:85 , srv318374192:85 , srv557324165:85 , srv693475259:85 , srv1060089958:86 , srv122899424:86 , srv1233204713:86 , srv1284327298:86 , srv1299478500:86 , srv1435682169:86 , srv1629907173:86 , srv1634490451:86 , srv1654632977:86 , srv1807335:86 , srv2029898661:86 , srv45940423:86 , srv495065549:86 , srv910965801:86 , srv1083446808:87 , srv1116334842:87 , srv1448493677:87 , srv1695055295:87 , srv1817248716:87 , srv1952145003:87 , srv1971923111:87 , srv1986363224:87 , srv2058167855:87 , srv316311702:87 , srv388382142:87 , srv605448350:87 , srv803726728:87 , srv833911551:87 , srv878714023:87 , srv99325228:87 , srv148563052:88 , srv1974758747:88 , srv296448398:88 , srv422128216:88 , srv58536521:88 , srv728474872:88 , srv940229362:88 , srv1175498467:89 , srv1305326789:89 , srv1964089380:89 , srv621029853:89 , srv64058751:89 , srv882011961:89 , srv944186205:89 , srv1530888195:90 , srv1586577997:91 , srv1955781188:91 , srv484846137:91 , srv2036170954:92 , srv642513965:92 , srv1493437973:93 , srv942083623:542 } 2023-10-20 10:55:34,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:34,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv45940423=69, srv944186205=98, srv318374192=65, srv878714023=90, srv1586577997=32, srv1853740115=48, srv388382142=66, srv1448493677=21, srv1629907173=33, srv1654632977=37, srv296448398=63, srv1636816046=35, srv1104759231=8, srv1701964014=39, srv316311702=64, srv904723070=93, srv1474352001=23, srv1299478500=15, srv1305326789=16, srv728474872=82, srv926487051=95, srv122899424=11, srv1484309726=24, srv1986363224=54, srv1116334842=9, srv754397315=83, srv1964089380=51, srv605448350=75, srv803726728=85, srv146664366=22, srv495065549=71, srv1233204713=12, srv1435794015=20, srv148563052=25, srv422128216=67, srv1060089958=4, srv1570931305=31, srv17715125=43, srv833911551=87, srv1952145003=49, srv2125799702=60, srv1971923111=52, srv1435682169=19, srv1396240289=18, srv1807335=46, srv1806412777=45, srv268236415=61, srv876075872=89, srv693475259=80, srv2062249717=58, srv58536521=74, srv1508507522=28, srv1955781188=50, srv642513965=79, srv1081426314=5, srv2058167855=57, srv484846137=70, srv557324165=72, srv1494687568=27, srv942083623=97, srv2036170954=56, srv2070113940=59, srv1770016068=42, srv1730518036=40, srv1012485467=0, srv882011961=91, srv1047725234=2, srv705654320=81, srv621029853=76, srv1545751096=30, srv17576380=41, srv894258434=92, srv1530888195=29, srv1974758747=53, srv2029898661=55, srv1646513189=36, srv1087769207=7, srv64058751=78, srv444177344=68, srv630541336=77, srv1351627939=17, srv1023786415=1, srv1053356277=3, srv1250178982=13, srv1083446808=6, srv1493437973=26, srv858903721=88, srv275969840=62, srv910965801=94, srv1634490451=34, srv1284327298=14, srv99325228=99, srv940229362=96, srv1817248716=47, srv1801527185=44, srv1695055295=38, srv808418427=86, srv1175498467=10, srv788520731=84, srv565706037=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:34,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:34,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:34,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:34,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:34,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:34,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:34,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 10:55:34,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 10:55:34,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 10:55:34,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 10:55:34,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 10:55:34,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 10:55:34,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 10:55:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 10:55:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 10:55:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 10:55:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 10:55:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 10:55:34,994 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0246355148534342); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039312269655344824, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0246355148534342); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:34,994 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 10:55:34,995 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015277715476119507, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0246355148534342); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039312269655344824, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0246355148534342); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 10:55:35,221 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 336 ms to try 5000 different iterations. Found a solution that moves 961 regions; Going from a computed imbalance of 0.015277715476119507 to a new imbalance of 6.742833114405139E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010455399369801359); MoveCostFunction : (multiplier=7.0, imbalance=0.10677777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:35,224 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv422128216:81 , srv484846137:81 , srv642513965:81 , srv1448493677:82 , srv1493437973:82 , srv876075872:83 , srv878714023:83 , srv1083446808:84 , srv1435682169:84 , srv1545751096:84 , srv1586577997:84 , srv1817248716:84 , srv1952145003:84 , srv1955781188:84 , srv1974758747:84 , srv2058167855:84 , srv296448398:84 , srv388382142:84 , srv45940423:84 , srv58536521:84 , srv64058751:84 , srv728474872:84 , srv833911551:84 , srv882011961:84 , srv1081426314:85 , srv1116334842:85 , srv1233204713:85 , srv1250178982:85 , srv1305326789:85 , srv1396240289:85 , srv1435794015:85 , srv1494687568:85 , srv1530888195:85 , srv1629907173:85 , srv1634490451:85 , srv1636816046:85 , srv1646513189:85 , srv17715125:85 , srv1801527185:85 , srv1964089380:85 , srv268236415:85 , srv605448350:85 , srv621029853:85 , srv705654320:85 , srv754397315:85 , srv803726728:85 , srv904723070:85 , srv910965801:85 , srv944186205:85 , srv99325228:85 , srv1053356277:86 , srv1060089958:86 , srv1175498467:86 , srv1284327298:86 , srv1570931305:86 , srv1701964014:86 , srv1853740115:86 , srv2029898661:86 , srv2036170954:86 , srv2062249717:86 , srv275969840:86 , srv495065549:86 , srv557324165:86 , srv693475259:86 , srv788520731:86 , srv894258434:86 , srv940229362:86 , srv1023786415:87 , srv1047725234:87 , srv1104759231:87 , srv122899424:87 , srv146664366:87 , srv1484309726:87 , srv148563052:87 , srv1654632977:87 , srv1695055295:87 , srv1730518036:87 , srv1806412777:87 , srv1807335:87 , srv1986363224:87 , srv2125799702:87 , srv316311702:87 , srv318374192:87 , srv565706037:87 , srv630541336:87 , srv808418427:87 , srv858903721:87 , srv926487051:87 , srv1012485467:88 , srv1087769207:88 , srv1299478500:88 , srv1351627939:88 , srv1474352001:88 , srv1508507522:88 , srv17576380:88 , srv1770016068:88 , srv444177344:88 , srv1971923111:89 , srv2070113940:89 , srv942083623:530 } 2023-10-20 10:55:35,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:35,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv45940423=69, srv944186205=98, srv318374192=65, srv878714023=90, srv1586577997=32, srv1853740115=48, srv388382142=66, srv1448493677=21, srv1629907173=33, srv1654632977=37, srv296448398=63, srv1636816046=35, srv1104759231=8, srv1701964014=39, srv316311702=64, srv904723070=93, srv1474352001=23, srv1299478500=15, srv1305326789=16, srv728474872=82, srv926487051=95, srv122899424=11, srv1484309726=24, srv1986363224=54, srv1116334842=9, srv754397315=83, srv1964089380=51, srv605448350=75, srv803726728=85, srv146664366=22, srv495065549=71, srv1233204713=12, srv1435794015=20, srv148563052=25, srv422128216=67, srv1060089958=4, srv1570931305=31, srv17715125=43, srv833911551=87, srv1952145003=49, srv2125799702=60, srv1971923111=52, srv1435682169=19, srv1396240289=18, srv1807335=46, srv1806412777=45, srv268236415=61, srv876075872=89, srv693475259=80, srv2062249717=58, srv58536521=74, srv1508507522=28, srv1955781188=50, srv642513965=79, srv1081426314=5, srv2058167855=57, srv484846137=70, srv557324165=72, srv1494687568=27, srv942083623=97, srv2036170954=56, srv2070113940=59, srv1770016068=42, srv1730518036=40, srv1012485467=0, srv882011961=91, srv1047725234=2, srv705654320=81, srv621029853=76, srv1545751096=30, srv17576380=41, srv894258434=92, srv1530888195=29, srv1974758747=53, srv2029898661=55, srv1646513189=36, srv1087769207=7, srv64058751=78, srv444177344=68, srv630541336=77, srv1351627939=17, srv1023786415=1, srv1053356277=3, srv1250178982=13, srv1083446808=6, srv1493437973=26, srv858903721=88, srv275969840=62, srv910965801=94, srv1634490451=34, srv1284327298=14, srv99325228=99, srv940229362=96, srv1817248716=47, srv1801527185=44, srv1695055295=38, srv808418427=86, srv1175498467=10, srv788520731=84, srv565706037=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:35,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:35,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:35,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:35,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:35,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:35,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 10:55:35,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 10:55:35,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 10:55:35,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 10:55:35,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 10:55:35,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 10:55:35,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 10:55:35,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 10:55:35,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010455399369801359); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:35,387 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 10:55:35,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.675542456173062E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010455399369801359); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 10:55:35,631 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 361 ms to try 5000 different iterations. Found a solution that moves 311 regions; Going from a computed imbalance of 6.675542456173062E-4 to a new imbalance of 5.098446859634876E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.883854646552481E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.034555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.883854646552481E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:35,631 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv630541336:82 , srv1299478500:83 , srv1351627939:83 , srv1448493677:83 , srv17576380:83 , srv1971923111:83 , srv1023786415:84 , srv1060089958:84 , srv1087769207:84 , srv1116334842:84 , srv1284327298:84 , srv146664366:84 , srv1494687568:84 , srv1508507522:84 , srv1530888195:84 , srv1629907173:84 , srv1654632977:84 , srv1770016068:84 , srv1806412777:84 , srv1807335:84 , srv1952145003:84 , srv1964089380:84 , srv1986363224:84 , srv2036170954:84 , srv2070113940:84 , srv2125799702:84 , srv268236415:84 , srv296448398:84 , srv45940423:84 , srv565706037:84 , srv58536521:84 , srv705654320:84 , srv788520731:84 , srv803726728:84 , srv808418427:84 , srv858903721:84 , srv878714023:84 , srv882011961:84 , srv904723070:84 , srv944186205:84 , srv1053356277:85 , srv1081426314:85 , srv1083446808:85 , srv1104759231:85 , srv1175498467:85 , srv122899424:85 , srv1233204713:85 , srv1250178982:85 , srv1305326789:85 , srv1396240289:85 , srv1435682169:85 , srv1435794015:85 , srv1484309726:85 , srv148563052:85 , srv1545751096:85 , srv1570931305:85 , srv1586577997:85 , srv1634490451:85 , srv1636816046:85 , srv1695055295:85 , srv1701964014:85 , srv1730518036:85 , srv1801527185:85 , srv1817248716:85 , srv1853740115:85 , srv1955781188:85 , srv1974758747:85 , srv2029898661:85 , srv2058167855:85 , srv2062249717:85 , srv316311702:85 , srv318374192:85 , srv388382142:85 , srv422128216:85 , srv495065549:85 , srv557324165:85 , srv605448350:85 , srv621029853:85 , srv64058751:85 , srv642513965:85 , srv693475259:85 , srv728474872:85 , srv754397315:85 , srv833911551:85 , srv910965801:85 , srv99325228:85 , srv1012485467:86 , srv1047725234:86 , srv1474352001:86 , srv1493437973:86 , srv1646513189:86 , srv17715125:86 , srv275969840:86 , srv444177344:86 , srv484846137:86 , srv876075872:86 , srv894258434:86 , srv926487051:86 , srv940229362:86 , srv942083623:619 } 2023-10-20 10:55:35,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:35,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv45940423=69, srv944186205=98, srv318374192=65, srv878714023=90, srv1586577997=32, srv1853740115=48, srv388382142=66, srv1448493677=21, srv1629907173=33, srv1654632977=37, srv296448398=63, srv1636816046=35, srv1104759231=8, srv1701964014=39, srv316311702=64, srv904723070=93, srv1474352001=23, srv1299478500=15, srv1305326789=16, srv728474872=82, srv926487051=95, srv122899424=11, srv1484309726=24, srv1986363224=54, srv1116334842=9, srv754397315=83, srv1964089380=51, srv605448350=75, srv803726728=85, srv146664366=22, srv495065549=71, srv1233204713=12, srv1435794015=20, srv148563052=25, srv422128216=67, srv1060089958=4, srv1570931305=31, srv17715125=43, srv833911551=87, srv1952145003=49, srv2125799702=60, srv1971923111=52, srv1435682169=19, srv1396240289=18, srv1807335=46, srv1806412777=45, srv268236415=61, srv876075872=89, srv693475259=80, srv2062249717=58, srv58536521=74, srv1508507522=28, srv1955781188=50, srv642513965=79, srv1081426314=5, srv2058167855=57, srv484846137=70, srv557324165=72, srv1494687568=27, srv942083623=97, srv2036170954=56, srv2070113940=59, srv1770016068=42, srv1730518036=40, srv1012485467=0, srv882011961=91, srv1047725234=2, srv705654320=81, srv621029853=76, srv1545751096=30, srv17576380=41, srv894258434=92, srv1530888195=29, srv1974758747=53, srv2029898661=55, srv1646513189=36, srv1087769207=7, srv64058751=78, srv444177344=68, srv630541336=77, srv1351627939=17, srv1023786415=1, srv1053356277=3, srv1250178982=13, srv1083446808=6, srv1493437973=26, srv858903721=88, srv275969840=62, srv910965801=94, srv1634490451=34, srv1284327298=14, srv99325228=99, srv940229362=96, srv1817248716=47, srv1801527185=44, srv1695055295=38, srv808418427=86, srv1175498467=10, srv788520731=84, srv565706037=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:35,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:35,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:35,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:35,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 10:55:35,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 10:55:35,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 10:55:35,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 10:55:35,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 10:55:35,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 10:55:35,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 10:55:35,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 10:55:35,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 10:55:35,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 10:55:35,751 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.883854646552481E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.883854646552481E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:35,751 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 10:55:35,751 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.8806800052105674E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.883854646552481E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.883854646552481E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 10:55:35,908 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 246 ms to try 5000 different iterations. Found a solution that moves 37 regions; Going from a computed imbalance of 4.8806800052105674E-5 to a new imbalance of 7.003148591249935E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:35,909 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1474352001:82 , srv1012485467:83 , srv1047725234:83 , srv1233204713:83 , srv1305326789:83 , srv1435794015:83 , srv1484309726:83 , srv1646513189:83 , srv1695055295:83 , srv1730518036:83 , srv444177344:83 , srv894258434:83 , srv910965801:83 , srv926487051:83 , srv940229362:83 , srv99325228:83 , srv1053356277:84 , srv1060089958:84 , srv1081426314:84 , srv1083446808:84 , srv1087769207:84 , srv1104759231:84 , srv1116334842:84 , srv1175498467:84 , srv122899424:84 , srv1284327298:84 , srv1299478500:84 , srv1396240289:84 , srv1435682169:84 , srv146664366:84 , srv148563052:84 , srv1493437973:84 , srv1508507522:84 , srv1545751096:84 , srv1570931305:84 , srv1586577997:84 , srv1634490451:84 , srv1636816046:84 , srv1654632977:84 , srv1701964014:84 , srv17576380:84 , srv1770016068:84 , srv17715125:84 , srv1801527185:84 , srv1806412777:84 , srv1807335:84 , srv1817248716:84 , srv1853740115:84 , srv1955781188:84 , srv1964089380:84 , srv1971923111:84 , srv1974758747:84 , srv2029898661:84 , srv2036170954:84 , srv2062249717:84 , srv2070113940:84 , srv2125799702:84 , srv268236415:84 , srv275969840:84 , srv316311702:84 , srv318374192:84 , srv388382142:84 , srv422128216:84 , srv45940423:84 , srv484846137:84 , srv495065549:84 , srv557324165:84 , srv565706037:84 , srv58536521:84 , srv605448350:84 , srv621029853:84 , srv64058751:84 , srv642513965:84 , srv693475259:84 , srv728474872:84 , srv754397315:84 , srv788520731:84 , srv808418427:84 , srv833911551:84 , srv876075872:84 , srv878714023:84 , srv882011961:84 , srv904723070:84 , srv1023786415:85 , srv1250178982:85 , srv1351627939:85 , srv1448493677:85 , srv1494687568:85 , srv1530888195:85 , srv1629907173:85 , srv1952145003:85 , srv1986363224:85 , srv2058167855:85 , srv296448398:85 , srv705654320:85 , srv803726728:85 , srv858903721:85 , srv944186205:85 , srv630541336:86 , srv942083623:684 } 2023-10-20 10:55:35,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:35,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv45940423=69, srv944186205=98, srv318374192=65, srv878714023=90, srv1586577997=32, srv1853740115=48, srv388382142=66, srv1448493677=21, srv1629907173=33, srv1654632977=37, srv296448398=63, srv1636816046=35, srv1104759231=8, srv1701964014=39, srv316311702=64, srv904723070=93, srv1474352001=23, srv1299478500=15, srv1305326789=16, srv728474872=82, srv926487051=95, srv122899424=11, srv1484309726=24, srv1986363224=54, srv1116334842=9, srv754397315=83, srv1964089380=51, srv605448350=75, srv803726728=85, srv146664366=22, srv495065549=71, srv1233204713=12, srv1435794015=20, srv148563052=25, srv422128216=67, srv1060089958=4, srv1570931305=31, srv17715125=43, srv833911551=87, srv1952145003=49, srv2125799702=60, srv1971923111=52, srv1435682169=19, srv1396240289=18, srv1807335=46, srv1806412777=45, srv268236415=61, srv876075872=89, srv693475259=80, srv2062249717=58, srv58536521=74, srv1508507522=28, srv1955781188=50, srv642513965=79, srv1081426314=5, srv2058167855=57, srv484846137=70, srv557324165=72, srv1494687568=27, srv942083623=97, srv2036170954=56, srv2070113940=59, srv1770016068=42, srv1730518036=40, srv1012485467=0, srv882011961=91, srv1047725234=2, srv705654320=81, srv621029853=76, srv1545751096=30, srv17576380=41, srv894258434=92, srv1530888195=29, srv1974758747=53, srv2029898661=55, srv1646513189=36, srv1087769207=7, srv64058751=78, srv444177344=68, srv630541336=77, srv1351627939=17, srv1023786415=1, srv1053356277=3, srv1250178982=13, srv1083446808=6, srv1493437973=26, srv858903721=88, srv275969840=62, srv910965801=94, srv1634490451=34, srv1284327298=14, srv99325228=99, srv940229362=96, srv1817248716=47, srv1801527185=44, srv1695055295=38, srv808418427=86, srv1175498467=10, srv788520731=84, srv565706037=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:35,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:35,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:35,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 10:55:35,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 10:55:35,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 10:55:35,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 10:55:35,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 10:55:35,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 10:55:36,035 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:36,036 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1474352001:82 , srv1012485467:83 , srv1047725234:83 , srv1233204713:83 , srv1305326789:83 , srv1435794015:83 , srv1484309726:83 , srv1646513189:83 , srv1695055295:83 , srv1730518036:83 , srv444177344:83 , srv894258434:83 , srv910965801:83 , srv926487051:83 , srv940229362:83 , srv99325228:83 , srv1053356277:84 , srv1060089958:84 , srv1081426314:84 , srv1083446808:84 , srv1087769207:84 , srv1104759231:84 , srv1116334842:84 , srv1175498467:84 , srv122899424:84 , srv1284327298:84 , srv1299478500:84 , srv1396240289:84 , srv1435682169:84 , srv146664366:84 , srv148563052:84 , srv1493437973:84 , srv1508507522:84 , srv1545751096:84 , srv1570931305:84 , srv1586577997:84 , srv1634490451:84 , srv1636816046:84 , srv1654632977:84 , srv1701964014:84 , srv17576380:84 , srv1770016068:84 , srv17715125:84 , srv1801527185:84 , srv1806412777:84 , srv1807335:84 , srv1817248716:84 , srv1853740115:84 , srv1955781188:84 , srv1964089380:84 , srv1971923111:84 , srv1974758747:84 , srv2029898661:84 , srv2036170954:84 , srv2062249717:84 , srv2070113940:84 , srv2125799702:84 , srv268236415:84 , srv275969840:84 , srv316311702:84 , srv318374192:84 , srv388382142:84 , srv422128216:84 , srv45940423:84 , srv484846137:84 , srv495065549:84 , srv557324165:84 , srv565706037:84 , srv58536521:84 , srv605448350:84 , srv621029853:84 , srv64058751:84 , srv642513965:84 , srv693475259:84 , srv728474872:84 , srv754397315:84 , srv788520731:84 , srv808418427:84 , srv833911551:84 , srv876075872:84 , srv878714023:84 , srv882011961:84 , srv904723070:84 , srv1023786415:85 , srv1250178982:85 , srv1351627939:85 , srv1448493677:85 , srv1494687568:85 , srv1530888195:85 , srv1629907173:85 , srv1952145003:85 , srv1986363224:85 , srv2058167855:85 , srv296448398:85 , srv705654320:85 , srv803726728:85 , srv858903721:85 , srv944186205:85 , srv630541336:86 , srv942083623:684 } 2023-10-20 10:55:36,076 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=434 (was 434), ProcessCount=177 (was 175) - ProcessCount LEAK? -, AvailableMemoryMB=2138 (was 2652) 2023-10-20 10:55:36,085 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=434, ProcessCount=177, AvailableMemoryMB=2137 2023-10-20 10:55:36,086 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-20 10:55:36,086 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-20 10:55:36,100 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023357998:84 , srv106939050:84 , srv122926496:84 , srv123089861:84 , srv1281683933:84 , srv1309105264:84 , srv1376188797:84 , srv1405142434:84 , srv1417175840:84 , srv1472865601:84 , srv1588166111:84 , srv1657182414:84 , srv174242356:84 , srv1888412117:84 , srv1914156467:84 , srv1921567864:84 , srv193025134:84 , srv201465971:84 , srv2108507115:84 , srv226618863:84 , srv255162580:84 , srv37312180:84 , srv437897550:84 , srv537235466:84 , srv564552858:84 , srv617829232:84 , srv690618251:84 , srv843081666:84 , srv961067316:84 , srv1332018174:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-20 10:55:36,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:55:36,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2108507115=19, srv1472865601=10, srv564552858=25, srv255162580=21, srv1914156467=15, srv1888412117=14, srv226618863=20, srv1023357998=0, srv1332018174=6, srv537235466=24, srv174242356=13, srv617829232=26, srv1417175840=9, srv106939050=1, srv1921567864=16, srv961067316=29, srv122926496=2, srv193025134=17, srv201465971=18, srv690618251=27, srv843081666=28, srv1376188797=7, srv1281683933=4, srv1405142434=8, srv37312180=22, srv1588166111=11, srv123089861=3, srv1309105264=5, srv1657182414=12, srv437897550=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-20 10:55:36,139 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1565049168916803, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:55:36,139 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912686185147049, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1565049168916803, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-20 10:57:36,121 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 8024051 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912686185147049 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:57:36,128 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1023357998:90 , srv106939050:90 , srv122926496:90 , srv123089861:90 , srv1281683933:90 , srv1309105264:90 , srv1332018174:90 , srv1376188797:90 , srv1405142434:90 , srv1417175840:90 , srv1472865601:90 , srv1588166111:90 , srv1657182414:90 , srv174242356:90 , srv1888412117:90 , srv1914156467:90 , srv1921567864:90 , srv193025134:90 , srv201465971:90 , srv2108507115:90 , srv226618863:90 , srv255162580:90 , srv37312180:90 , srv437897550:90 , srv537235466:90 , srv564552858:90 , srv617829232:90 , srv690618251:90 , srv843081666:90 , srv961067316:90 } 2023-10-20 10:57:36,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 10:57:36,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2108507115=19, srv1472865601=10, srv564552858=25, srv255162580=21, srv1914156467=15, srv1888412117=14, srv226618863=20, srv1023357998=0, srv1332018174=6, srv537235466=24, srv174242356=13, srv617829232=26, srv1417175840=9, srv106939050=1, srv1921567864=16, srv961067316=29, srv122926496=2, srv193025134=17, srv201465971=18, srv690618251=27, srv843081666=28, srv1376188797=7, srv1281683933=4, srv1405142434=8, srv37312180=22, srv1588166111=11, srv123089861=3, srv1309105264=5, srv1657182414=12, srv437897550=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 10:57:36,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 10:57:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 10:57:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-20 10:57:36,145 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 10:57:36,146 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1023357998:90 , srv106939050:90 , srv122926496:90 , srv123089861:90 , srv1281683933:90 , srv1309105264:90 , srv1332018174:90 , srv1376188797:90 , srv1405142434:90 , srv1417175840:90 , srv1472865601:90 , srv1588166111:90 , srv1657182414:90 , srv174242356:90 , srv1888412117:90 , srv1914156467:90 , srv1921567864:90 , srv193025134:90 , srv201465971:90 , srv2108507115:90 , srv226618863:90 , srv255162580:90 , srv37312180:90 , srv437897550:90 , srv537235466:90 , srv564552858:90 , srv617829232:90 , srv690618251:90 , srv843081666:90 , srv961067316:90 } 2023-10-20 10:57:36,157 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=425 (was 434), ProcessCount=174 (was 177), AvailableMemoryMB=699 (was 2137)