2024-01-27 18:53:14,343 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-27 18:53:14,657 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-27 18:53:14,774 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-27 18:53:14,843 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-27 18:53:14,852 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-27 18:53:14,865 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2549 2024-01-27 18:53:14,867 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-27 18:53:14,868 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-27 18:53:14,990 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018457664:84 , srv1119274909:84 , srv114218534:84 , srv1155951483:84 , srv1193266568:84 , srv1196974963:84 , srv1238230586:84 , srv1274927567:84 , srv1276759220:84 , srv1279973130:84 , srv128801732:84 , srv131348702:84 , srv1329522551:84 , srv1330558254:84 , srv1376022566:84 , srv1381949358:84 , srv1389340676:84 , srv1463557609:84 , srv1472560106:84 , srv1483991708:84 , srv1499801388:84 , srv1504115752:84 , srv152409252:84 , srv1534281618:84 , srv1554585097:84 , srv1555145827:84 , srv1562489461:84 , srv1571308894:84 , srv1596617313:84 , srv1604368824:84 , srv1617118085:84 , srv1659673476:84 , srv1717422904:84 , srv1725739303:84 , srv1732674505:84 , srv1745669956:84 , srv1783551495:84 , srv1792825661:84 , srv1801288664:84 , srv1807152952:84 , srv1815631354:84 , srv1914426197:84 , srv193630242:84 , srv1941862040:84 , srv1945901026:84 , srv1975218622:84 , srv2061353216:84 , srv2064631830:84 , srv209046795:84 , srv2104698641:84 , srv2110350465:84 , srv2111067639:84 , srv2115683:84 , srv2117048075:84 , srv2127789635:84 , srv220093431:84 , srv223323338:84 , srv240509206:84 , srv265613027:84 , srv286576584:84 , srv325867049:84 , srv327507911:84 , srv344200857:84 , srv415181679:84 , srv422200927:84 , srv436418864:84 , srv480787945:84 , srv493552557:84 , srv520402578:84 , srv546498111:84 , srv554585728:84 , srv587028920:84 , srv605230930:84 , srv631252316:84 , srv63456019:84 , srv672486766:84 , srv704676616:84 , srv710070832:84 , srv72140078:84 , srv723986622:84 , srv725123558:84 , srv742998251:84 , srv758965094:84 , srv763330398:84 , srv779419615:84 , srv821670206:84 , srv827800440:84 , srv83225885:84 , srv835913433:84 , srv850791636:84 , srv8781231:84 , srv88041684:84 , srv897691223:84 , srv927558755:84 , srv933650890:84 , srv941097135:84 , srv958692178:84 , srv958808154:84 , srv981849291:84 , srv1564417661:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-27 18:53:14,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:15,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv725123558=81, srv1329522551=12, srv546498111=70, srv1801288664=39, srv1483991708=19, srv1564417661=27, srv1193266568=4, srv1499801388=20, srv1330558254=13, srv344200857=63, srv827800440=87, srv821670206=86, srv209046795=49, srv1463557609=17, srv1815631354=41, srv1274927567=7, srv742998251=82, srv1504115752=21, srv1732674505=35, srv723986622=80, srv2061353216=47, srv704676616=77, srv958692178=97, srv152409252=22, srv605230930=73, srv1717422904=33, srv1659673476=32, srv1554585097=24, srv114218534=2, srv1945901026=45, srv63456019=75, srv672486766=76, srv1562489461=26, srv1941862040=44, srv325867049=61, srv1725739303=34, srv897691223=93, srv193630242=43, srv1745669956=36, srv2110350465=51, srv415181679=64, srv240509206=58, srv1018457664=0, srv1155951483=3, srv2064631830=48, srv1792825661=38, srv1389340676=16, srv1276759220=8, srv1604368824=30, srv1196974963=5, srv286576584=60, srv933650890=95, srv1571308894=28, srv1783551495=37, srv480787945=67, srv927558755=94, srv2104698641=50, srv941097135=96, srv710070832=78, srv835913433=89, srv758965094=83, srv83225885=88, srv131348702=11, srv436418864=66, srv554585728=71, srv763330398=84, srv8781231=91, srv493552557=68, srv88041684=92, srv1596617313=29, srv1555145827=25, srv2117048075=54, srv1238230586=6, srv265613027=59, srv1279973130=9, srv220093431=56, srv1119274909=1, srv1381949358=15, srv422200927=65, srv1617118085=31, srv958808154=98, srv2127789635=55, srv223323338=57, srv631252316=74, srv2111067639=52, srv1807152952=40, srv981849291=99, srv779419615=85, srv128801732=10, srv1975218622=46, srv1472560106=18, srv72140078=79, srv850791636=90, srv1534281618=23, srv2115683=53, srv587028920=72, srv1376022566=14, srv1914426197=42, srv520402578=69, srv327507911=62} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:15,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:15,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-27 18:53:15,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-27 18:53:15,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-27 18:53:15,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-27 18:53:15,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-27 18:53:15,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-27 18:53:15,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-27 18:53:15,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-27 18:53:15,344 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:15,344 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-27 18:53:15,345 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-27 18:53:15,894 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 770 ms to try 5000 different iterations. Found a solution that moves 4696 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1744780046492371. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.044698488310166776, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06085437346585237, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5217777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.044698488310166776, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:15,928 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv327507911:79 , srv672486766:80 , srv710070832:80 , srv1504115752:81 , srv2111067639:81 , srv631252316:81 , srv723986622:81 , srv958808154:81 , srv114218534:82 , srv1196974963:82 , srv1562489461:82 , srv325867049:82 , srv493552557:82 , srv1193266568:83 , srv1571308894:83 , srv2117048075:83 , srv240509206:83 , srv265613027:83 , srv742998251:83 , srv1018457664:84 , srv1238230586:84 , srv1472560106:84 , srv1483991708:84 , srv1792825661:84 , srv1801288664:84 , srv1807152952:84 , srv1815631354:84 , srv1975218622:84 , srv2061353216:84 , srv2127789635:84 , srv220093431:84 , srv605230930:84 , srv850791636:84 , srv927558755:84 , srv941097135:84 , srv958692178:84 , srv1276759220:85 , srv1330558254:85 , srv1659673476:85 , srv1745669956:85 , srv2064631830:85 , srv779419615:85 , srv933650890:85 , srv128801732:86 , srv1376022566:86 , srv1463557609:86 , srv1554585097:86 , srv1596617313:86 , srv223323338:86 , srv415181679:86 , srv436418864:86 , srv587028920:86 , srv72140078:86 , srv725123558:86 , srv758965094:86 , srv763330398:86 , srv827800440:86 , srv8781231:86 , srv88041684:86 , srv1155951483:87 , srv131348702:87 , srv1381949358:87 , srv1499801388:87 , srv1534281618:87 , srv1555145827:87 , srv1604368824:87 , srv1725739303:87 , srv1941862040:87 , srv1945901026:87 , srv2115683:87 , srv422200927:87 , srv83225885:87 , srv1119274909:88 , srv1617118085:88 , srv1914426197:88 , srv344200857:88 , srv821670206:88 , srv1389340676:89 , srv152409252:89 , srv1783551495:89 , srv193630242:89 , srv520402578:89 , srv554585728:89 , srv63456019:89 , srv1279973130:90 , srv1329522551:90 , srv1717422904:90 , srv1732674505:90 , srv209046795:90 , srv2110350465:90 , srv286576584:90 , srv704676616:90 , srv835913433:90 , srv2104698641:91 , srv480787945:91 , srv1274927567:92 , srv981849291:93 , srv546498111:94 , srv897691223:97 , srv1564417661:487 } 2024-01-27 18:53:15,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv725123558=81, srv1329522551=12, srv546498111=70, srv1801288664=39, srv1483991708=19, srv1564417661=27, srv1193266568=4, srv1499801388=20, srv1330558254=13, srv344200857=63, srv827800440=87, srv821670206=86, srv209046795=49, srv1463557609=17, srv1815631354=41, srv1274927567=7, srv742998251=82, srv1504115752=21, srv1732674505=35, srv723986622=80, srv2061353216=47, srv704676616=77, srv958692178=97, srv152409252=22, srv605230930=73, srv1717422904=33, srv1659673476=32, srv1554585097=24, srv114218534=2, srv1945901026=45, srv63456019=75, srv672486766=76, srv1562489461=26, srv1941862040=44, srv325867049=61, srv1725739303=34, srv897691223=93, srv193630242=43, srv1745669956=36, srv2110350465=51, srv415181679=64, srv240509206=58, srv1018457664=0, srv1155951483=3, srv2064631830=48, srv1792825661=38, srv1389340676=16, srv1276759220=8, srv1604368824=30, srv1196974963=5, srv286576584=60, srv933650890=95, srv1571308894=28, srv1783551495=37, srv480787945=67, srv927558755=94, srv2104698641=50, srv941097135=96, srv710070832=78, srv835913433=89, srv758965094=83, srv83225885=88, srv131348702=11, srv436418864=66, srv554585728=71, srv763330398=84, srv8781231=91, srv493552557=68, srv88041684=92, srv1596617313=29, srv1555145827=25, srv2117048075=54, srv1238230586=6, srv265613027=59, srv1279973130=9, srv220093431=56, srv1119274909=1, srv1381949358=15, srv422200927=65, srv1617118085=31, srv958808154=98, srv2127789635=55, srv223323338=57, srv631252316=74, srv2111067639=52, srv1807152952=40, srv981849291=99, srv779419615=85, srv128801732=10, srv1975218622=46, srv1472560106=18, srv72140078=79, srv850791636=90, srv1534281618=23, srv2115683=53, srv587028920=72, srv1376022566=14, srv1914426197=42, srv520402578=69, srv327507911=62} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-27 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-27 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-27 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-27 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-27 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-27 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-27 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-27 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-27 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-27 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-27 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-27 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-27 18:53:16,133 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.044698488310166776, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06085437346585237, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.044698488310166776, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:16,133 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-27 18:53:16,134 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.174445122554434, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.044698488310166776, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06085437346585237, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.044698488310166776, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-27 18:53:16,534 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 535 ms to try 5000 different iterations. Found a solution that moves 2658 regions; Going from a computed imbalance of 0.174445122554434 to a new imbalance of 0.014878144267001268. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02510533986102058, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03757521450677399, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29533333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02510533986102058, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:16,554 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv835913433:77 , srv2104698641:78 , srv1274927567:79 , srv1717422904:80 , srv897691223:80 , srv981849291:80 , srv209046795:81 , srv480787945:81 , srv704676616:81 , srv1279973130:82 , srv1381949358:82 , srv1389340676:82 , srv1783551495:82 , srv193630242:82 , srv2110350465:82 , srv546498111:82 , srv821670206:82 , srv1329522551:83 , srv1725739303:83 , srv1914426197:83 , srv1941862040:83 , srv2115683:83 , srv344200857:83 , srv1499801388:84 , srv1617118085:84 , srv1732674505:84 , srv1945901026:84 , srv554585728:84 , srv1155951483:85 , srv1276759220:85 , srv131348702:85 , srv1463557609:85 , srv152409252:85 , srv1534281618:85 , srv1554585097:85 , srv1815631354:85 , srv2064631830:85 , srv220093431:85 , srv286576584:85 , srv436418864:85 , srv520402578:85 , srv63456019:85 , srv8781231:85 , srv933650890:85 , srv128801732:86 , srv1376022566:86 , srv1555145827:86 , srv1604368824:86 , srv1659673476:86 , srv1745669956:86 , srv1807152952:86 , srv415181679:86 , srv587028920:86 , srv605230930:86 , srv72140078:86 , srv758965094:86 , srv827800440:86 , srv83225885:86 , srv88041684:86 , srv1119274909:87 , srv1196974963:87 , srv1483991708:87 , srv2061353216:87 , srv2127789635:87 , srv240509206:87 , srv422200927:87 , srv631252316:87 , srv941097135:87 , srv1018457664:88 , srv1504115752:88 , srv1596617313:88 , srv1801288664:88 , srv223323338:88 , srv265613027:88 , srv325867049:88 , srv493552557:88 , srv725123558:88 , srv763330398:88 , srv779419615:88 , srv958692178:88 , srv1193266568:89 , srv1238230586:89 , srv1330558254:89 , srv1571308894:89 , srv1792825661:89 , srv742998251:89 , srv850791636:89 , srv927558755:89 , srv114218534:90 , srv1472560106:90 , srv1975218622:90 , srv2111067639:90 , srv672486766:90 , srv710070832:90 , srv958808154:90 , srv723986622:91 , srv1562489461:92 , srv327507911:92 , srv2117048075:93 , srv1564417661:510 } 2024-01-27 18:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv725123558=81, srv1329522551=12, srv546498111=70, srv1801288664=39, srv1483991708=19, srv1564417661=27, srv1193266568=4, srv1499801388=20, srv1330558254=13, srv344200857=63, srv827800440=87, srv821670206=86, srv209046795=49, srv1463557609=17, srv1815631354=41, srv1274927567=7, srv742998251=82, srv1504115752=21, srv1732674505=35, srv723986622=80, srv2061353216=47, srv704676616=77, srv958692178=97, srv152409252=22, srv605230930=73, srv1717422904=33, srv1659673476=32, srv1554585097=24, srv114218534=2, srv1945901026=45, srv63456019=75, srv672486766=76, srv1562489461=26, srv1941862040=44, srv325867049=61, srv1725739303=34, srv897691223=93, srv193630242=43, srv1745669956=36, srv2110350465=51, srv415181679=64, srv240509206=58, srv1018457664=0, srv1155951483=3, srv2064631830=48, srv1792825661=38, srv1389340676=16, srv1276759220=8, srv1604368824=30, srv1196974963=5, srv286576584=60, srv933650890=95, srv1571308894=28, srv1783551495=37, srv480787945=67, srv927558755=94, srv2104698641=50, srv941097135=96, srv710070832=78, srv835913433=89, srv758965094=83, srv83225885=88, srv131348702=11, srv436418864=66, srv554585728=71, srv763330398=84, srv8781231=91, srv493552557=68, srv88041684=92, srv1596617313=29, srv1555145827=25, srv2117048075=54, srv1238230586=6, srv265613027=59, srv1279973130=9, srv220093431=56, srv1119274909=1, srv1381949358=15, srv422200927=65, srv1617118085=31, srv958808154=98, srv2127789635=55, srv223323338=57, srv631252316=74, srv2111067639=52, srv1807152952=40, srv981849291=99, srv779419615=85, srv128801732=10, srv1975218622=46, srv1472560106=18, srv72140078=79, srv850791636=90, srv1534281618=23, srv2115683=53, srv587028920=72, srv1376022566=14, srv1914426197=42, srv520402578=69, srv327507911=62} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:16,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:16,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:16,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:16,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:16,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:16,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-27 18:53:16,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-27 18:53:16,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-27 18:53:16,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-27 18:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-27 18:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-27 18:53:16,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-27 18:53:16,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-27 18:53:16,782 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02510533986102058, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03757521450677399, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02510533986102058, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:16,782 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-27 18:53:16,782 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014859532553205133, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02510533986102058, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03757521450677399, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02510533986102058, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-27 18:53:17,189 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 562 ms to try 5000 different iterations. Found a solution that moves 997 regions; Going from a computed imbalance of 0.014859532553205133 to a new imbalance of 6.660002745050188E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0077913942057109805); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011400277626721638); MoveCostFunction : (multiplier=7.0, imbalance=0.11077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0077913942057109805); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:17,193 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv897691223:81 , srv1119274909:82 , srv1596617313:82 , srv2117048075:82 , srv546498111:82 , srv725123558:82 , srv1732674505:83 , srv1975218622:83 , srv286576584:83 , srv520402578:83 , srv63456019:83 , srv763330398:83 , srv1193266568:84 , srv1329522551:84 , srv1330558254:84 , srv1472560106:84 , srv152409252:84 , srv1534281618:84 , srv1562489461:84 , srv1604368824:84 , srv223323338:84 , srv422200927:84 , srv554585728:84 , srv779419615:84 , srv83225885:84 , srv850791636:84 , srv981849291:84 , srv1018457664:85 , srv1155951483:85 , srv1238230586:85 , srv1274927567:85 , srv1279973130:85 , srv1376022566:85 , srv1463557609:85 , srv1554585097:85 , srv1555145827:85 , srv1792825661:85 , srv1801288664:85 , srv2111067639:85 , srv325867049:85 , srv415181679:85 , srv480787945:85 , srv587028920:85 , srv72140078:85 , srv827800440:85 , srv8781231:85 , srv88041684:85 , srv927558755:85 , srv941097135:85 , srv114218534:86 , srv128801732:86 , srv1499801388:86 , srv1571308894:86 , srv1617118085:86 , srv1725739303:86 , srv1745669956:86 , srv2061353216:86 , srv209046795:86 , srv2110350465:86 , srv2127789635:86 , srv265613027:86 , srv327507911:86 , srv436418864:86 , srv723986622:86 , srv758965094:86 , srv958692178:86 , srv958808154:86 , srv1276759220:87 , srv131348702:87 , srv1381949358:87 , srv1389340676:87 , srv1483991708:87 , srv1659673476:87 , srv1783551495:87 , srv1914426197:87 , srv193630242:87 , srv1945901026:87 , srv2064631830:87 , srv2115683:87 , srv240509206:87 , srv344200857:87 , srv493552557:87 , srv605230930:87 , srv704676616:87 , srv742998251:87 , srv933650890:87 , srv1196974963:88 , srv1717422904:88 , srv1807152952:88 , srv1815631354:88 , srv1941862040:88 , srv2104698641:88 , srv672486766:88 , srv710070832:88 , srv821670206:88 , srv1504115752:89 , srv220093431:89 , srv835913433:89 , srv631252316:91 , srv1564417661:530 } 2024-01-27 18:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv725123558=81, srv1329522551=12, srv546498111=70, srv1801288664=39, srv1483991708=19, srv1564417661=27, srv1193266568=4, srv1499801388=20, srv1330558254=13, srv344200857=63, srv827800440=87, srv821670206=86, srv209046795=49, srv1463557609=17, srv1815631354=41, srv1274927567=7, srv742998251=82, srv1504115752=21, srv1732674505=35, srv723986622=80, srv2061353216=47, srv704676616=77, srv958692178=97, srv152409252=22, srv605230930=73, srv1717422904=33, srv1659673476=32, srv1554585097=24, srv114218534=2, srv1945901026=45, srv63456019=75, srv672486766=76, srv1562489461=26, srv1941862040=44, srv325867049=61, srv1725739303=34, srv897691223=93, srv193630242=43, srv1745669956=36, srv2110350465=51, srv415181679=64, srv240509206=58, srv1018457664=0, srv1155951483=3, srv2064631830=48, srv1792825661=38, srv1389340676=16, srv1276759220=8, srv1604368824=30, srv1196974963=5, srv286576584=60, srv933650890=95, srv1571308894=28, srv1783551495=37, srv480787945=67, srv927558755=94, srv2104698641=50, srv941097135=96, srv710070832=78, srv835913433=89, srv758965094=83, srv83225885=88, srv131348702=11, srv436418864=66, srv554585728=71, srv763330398=84, srv8781231=91, srv493552557=68, srv88041684=92, srv1596617313=29, srv1555145827=25, srv2117048075=54, srv1238230586=6, srv265613027=59, srv1279973130=9, srv220093431=56, srv1119274909=1, srv1381949358=15, srv422200927=65, srv1617118085=31, srv958808154=98, srv2127789635=55, srv223323338=57, srv631252316=74, srv2111067639=52, srv1807152952=40, srv981849291=99, srv779419615=85, srv128801732=10, srv1975218622=46, srv1472560106=18, srv72140078=79, srv850791636=90, srv1534281618=23, srv2115683=53, srv587028920=72, srv1376022566=14, srv1914426197=42, srv520402578=69, srv327507911=62} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-27 18:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-27 18:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-27 18:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-27 18:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-27 18:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-27 18:53:17,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-27 18:53:17,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-27 18:53:17,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-27 18:53:17,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-27 18:53:17,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0077913942057109805); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011400277626721638); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0077913942057109805); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:17,387 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-27 18:53:17,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.590191312940529E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0077913942057109805); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011400277626721638); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0077913942057109805); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-27 18:53:17,702 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 451 ms to try 5000 different iterations. Found a solution that moves 291 regions; Going from a computed imbalance of 6.590191312940529E-4 to a new imbalance of 5.2575067600500254E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.03233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:17,703 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv131348702:83 , srv193630242:83 , srv631252316:83 , srv742998251:83 , srv1155951483:84 , srv1196974963:84 , srv1238230586:84 , srv128801732:84 , srv1330558254:84 , srv1389340676:84 , srv1483991708:84 , srv1504115752:84 , srv152409252:84 , srv1555145827:84 , srv1562489461:84 , srv1571308894:84 , srv1617118085:84 , srv1659673476:84 , srv1717422904:84 , srv1732674505:84 , srv1783551495:84 , srv1792825661:84 , srv1807152952:84 , srv1914426197:84 , srv1941862040:84 , srv1945901026:84 , srv1975218622:84 , srv2104698641:84 , srv2110350465:84 , srv2117048075:84 , srv220093431:84 , srv223323338:84 , srv286576584:84 , srv344200857:84 , srv554585728:84 , srv672486766:84 , srv704676616:84 , srv710070832:84 , srv723986622:84 , srv758965094:84 , srv821670206:84 , srv897691223:84 , srv927558755:84 , srv958692178:84 , srv1018457664:85 , srv1119274909:85 , srv114218534:85 , srv1276759220:85 , srv1279973130:85 , srv1329522551:85 , srv1376022566:85 , srv1472560106:85 , srv1499801388:85 , srv1604368824:85 , srv1801288664:85 , srv1815631354:85 , srv2061353216:85 , srv2064631830:85 , srv209046795:85 , srv2115683:85 , srv2127789635:85 , srv240509206:85 , srv265613027:85 , srv327507911:85 , srv415181679:85 , srv422200927:85 , srv436418864:85 , srv480787945:85 , srv493552557:85 , srv520402578:85 , srv546498111:85 , srv605230930:85 , srv63456019:85 , srv72140078:85 , srv725123558:85 , srv779419615:85 , srv827800440:85 , srv850791636:85 , srv88041684:85 , srv933650890:85 , srv958808154:85 , srv981849291:85 , srv1193266568:86 , srv1274927567:86 , srv1381949358:86 , srv1463557609:86 , srv1534281618:86 , srv1554585097:86 , srv1596617313:86 , srv1725739303:86 , srv1745669956:86 , srv2111067639:86 , srv587028920:86 , srv763330398:86 , srv83225885:86 , srv835913433:86 , srv8781231:86 , srv941097135:86 , srv325867049:87 , srv1564417661:615 } 2024-01-27 18:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv725123558=81, srv1329522551=12, srv546498111=70, srv1801288664=39, srv1483991708=19, srv1564417661=27, srv1193266568=4, srv1499801388=20, srv1330558254=13, srv344200857=63, srv827800440=87, srv821670206=86, srv209046795=49, srv1463557609=17, srv1815631354=41, srv1274927567=7, srv742998251=82, srv1504115752=21, srv1732674505=35, srv723986622=80, srv2061353216=47, srv704676616=77, srv958692178=97, srv152409252=22, srv605230930=73, srv1717422904=33, srv1659673476=32, srv1554585097=24, srv114218534=2, srv1945901026=45, srv63456019=75, srv672486766=76, srv1562489461=26, srv1941862040=44, srv325867049=61, srv1725739303=34, srv897691223=93, srv193630242=43, srv1745669956=36, srv2110350465=51, srv415181679=64, srv240509206=58, srv1018457664=0, srv1155951483=3, srv2064631830=48, srv1792825661=38, srv1389340676=16, srv1276759220=8, srv1604368824=30, srv1196974963=5, srv286576584=60, srv933650890=95, srv1571308894=28, srv1783551495=37, srv480787945=67, srv927558755=94, srv2104698641=50, srv941097135=96, srv710070832=78, srv835913433=89, srv758965094=83, srv83225885=88, srv131348702=11, srv436418864=66, srv554585728=71, srv763330398=84, srv8781231=91, srv493552557=68, srv88041684=92, srv1596617313=29, srv1555145827=25, srv2117048075=54, srv1238230586=6, srv265613027=59, srv1279973130=9, srv220093431=56, srv1119274909=1, srv1381949358=15, srv422200927=65, srv1617118085=31, srv958808154=98, srv2127789635=55, srv223323338=57, srv631252316=74, srv2111067639=52, srv1807152952=40, srv981849291=99, srv779419615=85, srv128801732=10, srv1975218622=46, srv1472560106=18, srv72140078=79, srv850791636=90, srv1534281618=23, srv2115683=53, srv587028920=72, srv1376022566=14, srv1914426197=42, srv520402578=69, srv327507911=62} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-27 18:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-27 18:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-27 18:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-27 18:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-27 18:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-27 18:53:17,883 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:17,884 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-27 18:53:17,884 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.0537442049456075E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-27 18:53:18,193 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 444 ms to try 5000 different iterations. Found a solution that moves 43 regions; Going from a computed imbalance of 5.0537442049456075E-5 to a new imbalance of 7.374331731707109E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0047777777777777775); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:18,193 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv114218534:83 , srv1472560106:83 , srv1745669956:83 , srv422200927:83 , srv546498111:83 , srv587028920:83 , srv763330398:83 , srv83225885:83 , srv1018457664:84 , srv1119274909:84 , srv1193266568:84 , srv1238230586:84 , srv1274927567:84 , srv1276759220:84 , srv1279973130:84 , srv128801732:84 , srv131348702:84 , srv1329522551:84 , srv1330558254:84 , srv1376022566:84 , srv1381949358:84 , srv1389340676:84 , srv1463557609:84 , srv1483991708:84 , srv1499801388:84 , srv1504115752:84 , srv152409252:84 , srv1534281618:84 , srv1554585097:84 , srv1555145827:84 , srv1562489461:84 , srv1571308894:84 , srv1596617313:84 , srv1604368824:84 , srv1617118085:84 , srv1659673476:84 , srv1717422904:84 , srv1725739303:84 , srv1783551495:84 , srv1792825661:84 , srv1801288664:84 , srv1807152952:84 , srv1815631354:84 , srv1914426197:84 , srv1941862040:84 , srv1945901026:84 , srv2061353216:84 , srv2064631830:84 , srv209046795:84 , srv2110350465:84 , srv2111067639:84 , srv2115683:84 , srv2117048075:84 , srv2127789635:84 , srv220093431:84 , srv223323338:84 , srv240509206:84 , srv265613027:84 , srv286576584:84 , srv325867049:84 , srv327507911:84 , srv344200857:84 , srv415181679:84 , srv436418864:84 , srv480787945:84 , srv493552557:84 , srv520402578:84 , srv605230930:84 , srv631252316:84 , srv63456019:84 , srv672486766:84 , srv704676616:84 , srv710070832:84 , srv72140078:84 , srv723986622:84 , srv742998251:84 , srv758965094:84 , srv779419615:84 , srv821670206:84 , srv827800440:84 , srv835913433:84 , srv850791636:84 , srv8781231:84 , srv88041684:84 , srv897691223:84 , srv927558755:84 , srv933650890:84 , srv941097135:84 , srv958692178:84 , srv958808154:84 , srv981849291:84 , srv1155951483:85 , srv1196974963:85 , srv1732674505:85 , srv193630242:85 , srv1975218622:85 , srv2104698641:85 , srv554585728:85 , srv725123558:85 , srv1564417661:684 } 2024-01-27 18:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv725123558=81, srv1329522551=12, srv546498111=70, srv1801288664=39, srv1483991708=19, srv1564417661=27, srv1193266568=4, srv1499801388=20, srv1330558254=13, srv344200857=63, srv827800440=87, srv821670206=86, srv209046795=49, srv1463557609=17, srv1815631354=41, srv1274927567=7, srv742998251=82, srv1504115752=21, srv1732674505=35, srv723986622=80, srv2061353216=47, srv704676616=77, srv958692178=97, srv152409252=22, srv605230930=73, srv1717422904=33, srv1659673476=32, srv1554585097=24, srv114218534=2, srv1945901026=45, srv63456019=75, srv672486766=76, srv1562489461=26, srv1941862040=44, srv325867049=61, srv1725739303=34, srv897691223=93, srv193630242=43, srv1745669956=36, srv2110350465=51, srv415181679=64, srv240509206=58, srv1018457664=0, srv1155951483=3, srv2064631830=48, srv1792825661=38, srv1389340676=16, srv1276759220=8, srv1604368824=30, srv1196974963=5, srv286576584=60, srv933650890=95, srv1571308894=28, srv1783551495=37, srv480787945=67, srv927558755=94, srv2104698641=50, srv941097135=96, srv710070832=78, srv835913433=89, srv758965094=83, srv83225885=88, srv131348702=11, srv436418864=66, srv554585728=71, srv763330398=84, srv8781231=91, srv493552557=68, srv88041684=92, srv1596617313=29, srv1555145827=25, srv2117048075=54, srv1238230586=6, srv265613027=59, srv1279973130=9, srv220093431=56, srv1119274909=1, srv1381949358=15, srv422200927=65, srv1617118085=31, srv958808154=98, srv2127789635=55, srv223323338=57, srv631252316=74, srv2111067639=52, srv1807152952=40, srv981849291=99, srv779419615=85, srv128801732=10, srv1975218622=46, srv1472560106=18, srv72140078=79, srv850791636=90, srv1534281618=23, srv2115683=53, srv587028920=72, srv1376022566=14, srv1914426197=42, srv520402578=69, srv327507911=62} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:18,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:18,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:18,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:18,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:18,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:18,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:18,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:18,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:18,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:18,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-27 18:53:18,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-27 18:53:18,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-27 18:53:18,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-27 18:53:18,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-27 18:53:18,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-27 18:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-27 18:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-27 18:53:18,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-27 18:53:18,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-27 18:53:18,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-27 18:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-27 18:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-27 18:53:18,383 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:18,384 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv114218534:83 , srv1472560106:83 , srv1745669956:83 , srv422200927:83 , srv546498111:83 , srv587028920:83 , srv763330398:83 , srv83225885:83 , srv1018457664:84 , srv1119274909:84 , srv1193266568:84 , srv1238230586:84 , srv1274927567:84 , srv1276759220:84 , srv1279973130:84 , srv128801732:84 , srv131348702:84 , srv1329522551:84 , srv1330558254:84 , srv1376022566:84 , srv1381949358:84 , srv1389340676:84 , srv1463557609:84 , srv1483991708:84 , srv1499801388:84 , srv1504115752:84 , srv152409252:84 , srv1534281618:84 , srv1554585097:84 , srv1555145827:84 , srv1562489461:84 , srv1571308894:84 , srv1596617313:84 , srv1604368824:84 , srv1617118085:84 , srv1659673476:84 , srv1717422904:84 , srv1725739303:84 , srv1783551495:84 , srv1792825661:84 , srv1801288664:84 , srv1807152952:84 , srv1815631354:84 , srv1914426197:84 , srv1941862040:84 , srv1945901026:84 , srv2061353216:84 , srv2064631830:84 , srv209046795:84 , srv2110350465:84 , srv2111067639:84 , srv2115683:84 , srv2117048075:84 , srv2127789635:84 , srv220093431:84 , srv223323338:84 , srv240509206:84 , srv265613027:84 , srv286576584:84 , srv325867049:84 , srv327507911:84 , srv344200857:84 , srv415181679:84 , srv436418864:84 , srv480787945:84 , srv493552557:84 , srv520402578:84 , srv605230930:84 , srv631252316:84 , srv63456019:84 , srv672486766:84 , srv704676616:84 , srv710070832:84 , srv72140078:84 , srv723986622:84 , srv742998251:84 , srv758965094:84 , srv779419615:84 , srv821670206:84 , srv827800440:84 , srv835913433:84 , srv850791636:84 , srv8781231:84 , srv88041684:84 , srv897691223:84 , srv927558755:84 , srv933650890:84 , srv941097135:84 , srv958692178:84 , srv958808154:84 , srv981849291:84 , srv1155951483:85 , srv1196974963:85 , srv1732674505:85 , srv193630242:85 , srv1975218622:85 , srv2104698641:85 , srv554585728:85 , srv725123558:85 , srv1564417661:684 } 2024-01-27 18:53:18,419 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=303 (was 277) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=1589 (was 2549) 2024-01-27 18:53:18,429 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=303, ProcessCount=167, AvailableMemoryMB=1588 2024-01-27 18:53:18,429 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-27 18:53:18,430 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-27 18:53:18,446 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002389453:84 , srv1030001327:84 , srv1068914535:84 , srv1082786566:84 , srv1092659213:84 , srv113317950:84 , srv1202583960:84 , srv1214387629:84 , srv1241064266:84 , srv1296141636:84 , srv1519227440:84 , srv1521852437:84 , srv1572072597:84 , srv1600618835:84 , srv1634304305:84 , srv1659046656:84 , srv1798441195:84 , srv1874625115:84 , srv1897828989:84 , srv1917020654:84 , srv1917849488:84 , srv1974330544:84 , srv1990477137:84 , srv278144241:84 , srv311864915:84 , srv367066985:84 , srv458433133:84 , srv481448993:84 , srv52963416:84 , srv2132601851:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-27 18:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1202583960=6, srv1521852437=11, srv1659046656=15, srv1241064266=8, srv1600618835=13, srv311864915=25, srv1917020654=19, srv52963416=29, srv1897828989=18, srv1917849488=20, srv1002389453=0, srv2132601851=23, srv481448993=28, srv1634304305=14, srv1974330544=21, srv1068914535=2, srv1092659213=4, srv1214387629=7, srv1572072597=12, srv1030001327=1, srv1798441195=16, srv1990477137=22, srv367066985=26, srv1296141636=9, srv113317950=5, srv278144241=24, srv1082786566=3, srv1519227440=10, srv1874625115=17, srv458433133=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-27 18:53:18,488 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0492935616820864, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:53:18,488 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912348365667558, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0492935616820864, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-27 18:55:18,474 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6240562 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912348365667558 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:55:18,480 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1002389453:90 , srv1030001327:90 , srv1068914535:90 , srv1082786566:90 , srv1092659213:90 , srv113317950:90 , srv1202583960:90 , srv1214387629:90 , srv1241064266:90 , srv1296141636:90 , srv1519227440:90 , srv1521852437:90 , srv1572072597:90 , srv1600618835:90 , srv1634304305:90 , srv1659046656:90 , srv1798441195:90 , srv1874625115:90 , srv1897828989:90 , srv1917020654:90 , srv1917849488:90 , srv1974330544:90 , srv1990477137:90 , srv2132601851:90 , srv278144241:90 , srv311864915:90 , srv367066985:90 , srv458433133:90 , srv481448993:90 , srv52963416:90 } 2024-01-27 18:55:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-27 18:55:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1202583960=6, srv1521852437=11, srv1659046656=15, srv1241064266=8, srv1600618835=13, srv311864915=25, srv1917020654=19, srv52963416=29, srv1897828989=18, srv1917849488=20, srv1002389453=0, srv2132601851=23, srv481448993=28, srv1634304305=14, srv1974330544=21, srv1068914535=2, srv1092659213=4, srv1214387629=7, srv1572072597=12, srv1030001327=1, srv1798441195=16, srv1990477137=22, srv367066985=26, srv1296141636=9, srv113317950=5, srv278144241=24, srv1082786566=3, srv1519227440=10, srv1874625115=17, srv458433133=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-27 18:55:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-27 18:55:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-27 18:55:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-27 18:55:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-27 18:55:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-27 18:55:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-27 18:55:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-27 18:55:18,498 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-27 18:55:18,498 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1002389453:90 , srv1030001327:90 , srv1068914535:90 , srv1082786566:90 , srv1092659213:90 , srv113317950:90 , srv1202583960:90 , srv1214387629:90 , srv1241064266:90 , srv1296141636:90 , srv1519227440:90 , srv1521852437:90 , srv1572072597:90 , srv1600618835:90 , srv1634304305:90 , srv1659046656:90 , srv1798441195:90 , srv1874625115:90 , srv1897828989:90 , srv1917020654:90 , srv1917849488:90 , srv1974330544:90 , srv1990477137:90 , srv2132601851:90 , srv278144241:90 , srv311864915:90 , srv367066985:90 , srv458433133:90 , srv481448993:90 , srv52963416:90 } 2024-01-27 18:55:18,512 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 303), ProcessCount=167 (was 167), AvailableMemoryMB=999 (was 1588)