2023-08-02 11:53:09,977 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-02 11:53:10,283 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-02 11:53:10,402 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-02 11:53:10,463 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-02 11:53:10,470 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-02 11:53:10,486 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=309, ProcessCount=174, AvailableMemoryMB=2922 2023-08-02 11:53:10,488 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-02 11:53:10,489 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-02 11:53:10,616 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006678645:84 , srv1010013152:84 , srv1042027882:84 , srv1055709457:84 , srv1082862180:84 , srv1143925668:84 , srv116316770:84 , srv1164830434:84 , srv1168064955:84 , srv1183749223:84 , srv1212429541:84 , srv1263447659:84 , srv1272564469:84 , srv1296065405:84 , srv1322858487:84 , srv1322935552:84 , srv1397098656:84 , srv1424365943:84 , srv1427689385:84 , srv1443539513:84 , srv1451449700:84 , srv1482270877:84 , srv150052848:84 , srv1504144420:84 , srv1526983172:84 , srv1647850907:84 , srv1670607822:84 , srv1683177602:84 , srv1703831152:84 , srv1744516773:84 , srv1786373837:84 , srv178739509:84 , srv1788900439:84 , srv1791166257:84 , srv1796390744:84 , srv1804691720:84 , srv1805431338:84 , srv1821749726:84 , srv1823342454:84 , srv1864352040:84 , srv1879872708:84 , srv1941372948:84 , srv1944372834:84 , srv19607250:84 , srv1967445334:84 , srv1973685291:84 , srv2000607841:84 , srv2022084517:84 , srv2029601748:84 , srv2034712193:84 , srv203818760:84 , srv2044166373:84 , srv2044968456:84 , srv2067934561:84 , srv2081075314:84 , srv2092877441:84 , srv2096396523:84 , srv2119302110:84 , srv2119463155:84 , srv2137320294:84 , srv2145937396:84 , srv220631252:84 , srv253289520:84 , srv264643384:84 , srv282981611:84 , srv283122155:84 , srv297330025:84 , srv316508715:84 , srv330135278:84 , srv360987629:84 , srv374689516:84 , srv403359514:84 , srv407220570:84 , srv410824557:84 , srv470921688:84 , srv471583206:84 , srv555399416:84 , srv575756832:84 , srv579613773:84 , srv607022969:84 , srv608958763:84 , srv610226431:84 , srv621282564:84 , srv641680021:84 , srv644095809:84 , srv711011958:84 , srv713797429:84 , srv798622233:84 , srv82939676:84 , srv83002626:84 , srv886281159:84 , srv899605184:84 , srv905883941:84 , srv908134147:84 , srv911858924:84 , srv943233995:84 , srv970178054:84 , srv983031256:84 , srv993492736:84 , srv962428461:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-02 11:53:10,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:10,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv253289520=62, srv316508715=67, srv150052848=22, srv403359514=71, srv1967445334=44, srv1010013152=1, srv1424365943=17, srv1143925668=5, srv1823342454=38, srv203818760=50, srv116316770=6, srv178739509=31, srv82939676=88, srv1427689385=18, srv1941372948=41, srv2145937396=60, srv1744516773=29, srv1164830434=7, srv1443539513=19, srv2067934561=53, srv1944372834=42, srv2092877441=55, srv297330025=66, srv1805431338=36, srv1263447659=11, srv1647850907=25, srv2044968456=52, srv1006678645=0, srv264643384=63, srv471583206=75, srv983031256=98, srv1042027882=2, srv1804691720=35, srv2044166373=51, srv1397098656=16, srv1973685291=45, srv2137320294=59, srv899605184=91, srv2034712193=49, srv1082862180=4, srv1670607822=26, srv886281159=90, srv2119463155=58, srv360987629=69, srv575756832=77, srv1055709457=3, srv607022969=79, srv1526983172=24, srv1791166257=33, srv644095809=84, srv1482270877=21, srv610226431=81, srv1322858487=14, srv1864352040=39, srv905883941=92, srv555399416=76, srv220631252=61, srv993492736=99, srv2000607841=46, srv1788900439=32, srv2081075314=54, srv711011958=85, srv1212429541=10, srv374689516=70, srv943233995=95, srv1451449700=20, srv19607250=43, srv2022084517=47, srv330135278=68, srv470921688=74, srv911858924=94, srv282981611=64, srv1322935552=15, srv579613773=78, srv713797429=86, srv410824557=73, srv2029601748=48, srv641680021=83, srv1786373837=30, srv908134147=93, srv962428461=96, srv1821749726=37, srv621282564=82, srv1296065405=13, srv1683177602=27, srv1168064955=8, srv283122155=65, srv1504144420=23, srv2119302110=57, srv798622233=87, srv1183749223=9, srv608958763=80, srv1879872708=40, srv1703831152=28, srv83002626=89, srv407220570=72, srv2096396523=56, srv970178054=97, srv1272564469=12, srv1796390744=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:10,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:10,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:10,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:10,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:10,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:10,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:10,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:10,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:10,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:10,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:10,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:10,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:10,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:10,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:10,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:10,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:10,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:10,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:10,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:10,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:10,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:10,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:10,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:10,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:10,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-02 11:53:10,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-02 11:53:10,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-02 11:53:10,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-02 11:53:10,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-02 11:53:10,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-02 11:53:10,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-02 11:53:10,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-02 11:53:10,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-02 11:53:10,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-02 11:53:10,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-02 11:53:10,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-02 11:53:10,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-02 11:53:10,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-02 11:53:10,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-02 11:53:10,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-02 11:53:10,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-02 11:53:10,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-02 11:53:10,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-02 11:53:10,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-02 11:53:10,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-02 11:53:10,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-02 11:53:10,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-02 11:53:10,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-02 11:53:10,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-02 11:53:10,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-02 11:53:10,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-02 11:53:10,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-02 11:53:10,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-02 11:53:10,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-02 11:53:10,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-02 11:53:10,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-02 11:53:10,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-02 11:53:10,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-02 11:53:10,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-02 11:53:10,934 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:10,935 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-02 11:53:10,935 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-02 11:53:11,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 492 ms to try 5000 different iterations. Found a solution that moves 4726 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17680383854846213. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04210687771334902, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05527403418621491, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5251111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04210687771334902, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.236, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:11,287 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1804691720:80 , srv1670607822:81 , srv2119463155:81 , srv316508715:81 , srv283122155:82 , srv407220570:82 , srv471583206:82 , srv607022969:82 , srv993492736:82 , srv1164830434:83 , srv1263447659:83 , srv1322858487:83 , srv19607250:83 , srv2067934561:83 , srv360987629:83 , srv621282564:83 , srv899605184:83 , srv970178054:83 , srv1042027882:84 , srv1786373837:84 , srv178739509:84 , srv1796390744:84 , srv2022084517:84 , srv2081075314:84 , srv610226431:84 , srv641680021:84 , srv1010013152:85 , srv1143925668:85 , srv1427689385:85 , srv150052848:85 , srv1683177602:85 , srv1791166257:85 , srv1823342454:85 , srv1864352040:85 , srv1944372834:85 , srv2029601748:85 , srv410824557:85 , srv470921688:85 , srv555399416:85 , srv644095809:85 , srv713797429:85 , srv83002626:85 , srv886281159:85 , srv943233995:85 , srv1055709457:86 , srv116316770:86 , srv1272564469:86 , srv1647850907:86 , srv1941372948:86 , srv1967445334:86 , srv2044166373:86 , srv2044968456:86 , srv253289520:86 , srv282981611:86 , srv575756832:86 , srv579613773:86 , srv798622233:86 , srv905883941:86 , srv911858924:86 , srv1183749223:87 , srv1397098656:87 , srv1504144420:87 , srv1526983172:87 , srv1805431338:87 , srv2119302110:87 , srv2145937396:87 , srv297330025:87 , srv82939676:87 , srv1006678645:88 , srv1212429541:88 , srv1424365943:88 , srv1451449700:88 , srv1744516773:88 , srv1879872708:88 , srv2000607841:88 , srv203818760:88 , srv2092877441:88 , srv330135278:88 , srv374689516:88 , srv403359514:88 , srv711011958:88 , srv908134147:88 , srv983031256:88 , srv1296065405:89 , srv220631252:89 , srv264643384:89 , srv1443539513:90 , srv1821749726:90 , srv2137320294:90 , srv1168064955:91 , srv1322935552:91 , srv2034712193:91 , srv1082862180:92 , srv1788900439:92 , srv2096396523:92 , srv1703831152:93 , srv608958763:93 , srv1482270877:94 , srv1973685291:95 , srv962428461:464 } 2023-08-02 11:53:11,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:11,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv253289520=62, srv316508715=67, srv150052848=22, srv403359514=71, srv1967445334=44, srv1010013152=1, srv1424365943=17, srv1143925668=5, srv1823342454=38, srv203818760=50, srv116316770=6, srv178739509=31, srv82939676=88, srv1427689385=18, srv1941372948=41, srv2145937396=60, srv1744516773=29, srv1164830434=7, srv1443539513=19, srv2067934561=53, srv1944372834=42, srv2092877441=55, srv297330025=66, srv1805431338=36, srv1263447659=11, srv1647850907=25, srv2044968456=52, srv1006678645=0, srv264643384=63, srv471583206=75, srv983031256=98, srv1042027882=2, srv1804691720=35, srv2044166373=51, srv1397098656=16, srv1973685291=45, srv2137320294=59, srv899605184=91, srv2034712193=49, srv1082862180=4, srv1670607822=26, srv886281159=90, srv2119463155=58, srv360987629=69, srv575756832=77, srv1055709457=3, srv607022969=79, srv1526983172=24, srv1791166257=33, srv644095809=84, srv1482270877=21, srv610226431=81, srv1322858487=14, srv1864352040=39, srv905883941=92, srv555399416=76, srv220631252=61, srv993492736=99, srv2000607841=46, srv1788900439=32, srv2081075314=54, srv711011958=85, srv1212429541=10, srv374689516=70, srv943233995=95, srv1451449700=20, srv19607250=43, srv2022084517=47, srv330135278=68, srv470921688=74, srv911858924=94, srv282981611=64, srv1322935552=15, srv579613773=78, srv713797429=86, srv410824557=73, srv2029601748=48, srv641680021=83, srv1786373837=30, srv908134147=93, srv962428461=96, srv1821749726=37, srv621282564=82, srv1296065405=13, srv1683177602=27, srv1168064955=8, srv283122155=65, srv1504144420=23, srv2119302110=57, srv798622233=87, srv1183749223=9, srv608958763=80, srv1879872708=40, srv1703831152=28, srv83002626=89, srv407220570=72, srv2096396523=56, srv970178054=97, srv1272564469=12, srv1796390744=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:11,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:11,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:11,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-02 11:53:11,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-02 11:53:11,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-02 11:53:11,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-02 11:53:11,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-02 11:53:11,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-02 11:53:11,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-02 11:53:11,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-02 11:53:11,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-02 11:53:11,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-02 11:53:11,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-02 11:53:11,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-02 11:53:11,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-02 11:53:11,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04210687771334902, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05527403418621491, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04210687771334902, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.236, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:11,502 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-02 11:53:11,502 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17677074638916923, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04210687771334902, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05527403418621491, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04210687771334902, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.236, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-02 11:53:11,910 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 559 ms to try 5000 different iterations. Found a solution that moves 2503 regions; Going from a computed imbalance of 0.17677074638916923 to a new imbalance of 0.015858268138652397. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024073297060757656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03499238333028429, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2781111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024073297060757656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:11,928 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1482270877:76 , srv1788900439:77 , srv608958763:77 , srv1973685291:78 , srv2034712193:78 , srv1703831152:80 , srv220631252:81 , srv1168064955:82 , srv1879872708:82 , srv2096396523:82 , srv2137320294:82 , srv1006678645:83 , srv1055709457:83 , srv1212429541:83 , srv1322935552:83 , srv1451449700:83 , srv1821749726:83 , srv253289520:83 , srv264643384:83 , srv82939676:83 , srv983031256:83 , srv1082862180:84 , srv1272564469:84 , srv1296065405:84 , srv1424365943:84 , srv1443539513:84 , srv1744516773:84 , srv2000607841:84 , srv203818760:84 , srv2119302110:84 , srv330135278:84 , srv374689516:84 , srv403359514:84 , srv908134147:84 , srv1183749223:85 , srv1397098656:85 , srv1504144420:85 , srv1526983172:85 , srv1941372948:85 , srv2092877441:85 , srv555399416:85 , srv575756832:85 , srv644095809:85 , srv711011958:85 , srv798622233:85 , srv1042027882:86 , srv1647850907:86 , srv1805431338:86 , srv1864352040:86 , srv2044968456:86 , srv2145937396:86 , srv282981611:86 , srv283122155:86 , srv579613773:86 , srv641680021:86 , srv713797429:86 , srv886281159:86 , srv905883941:86 , srv911858924:86 , srv1143925668:87 , srv1263447659:87 , srv1427689385:87 , srv1791166257:87 , srv1823342454:87 , srv1944372834:87 , srv2022084517:87 , srv2029601748:87 , srv2081075314:87 , srv297330025:87 , srv360987629:87 , srv410824557:87 , srv470921688:87 , srv610226431:87 , srv83002626:87 , srv943233995:87 , srv970178054:87 , srv116316770:88 , srv1683177602:88 , srv1786373837:88 , srv178739509:88 , srv899605184:88 , srv1010013152:89 , srv1164830434:89 , srv1322858487:89 , srv150052848:89 , srv1796390744:89 , srv19607250:89 , srv2044166373:89 , srv621282564:89 , srv1804691720:90 , srv1967445334:90 , srv2067934561:90 , srv471583206:90 , srv1670607822:91 , srv2119463155:91 , srv407220570:91 , srv607022969:91 , srv993492736:91 , srv316508715:94 , srv962428461:524 } 2023-08-02 11:53:11,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:11,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv253289520=62, srv316508715=67, srv150052848=22, srv403359514=71, srv1967445334=44, srv1010013152=1, srv1424365943=17, srv1143925668=5, srv1823342454=38, srv203818760=50, srv116316770=6, srv178739509=31, srv82939676=88, srv1427689385=18, srv1941372948=41, srv2145937396=60, srv1744516773=29, srv1164830434=7, srv1443539513=19, srv2067934561=53, srv1944372834=42, srv2092877441=55, srv297330025=66, srv1805431338=36, srv1263447659=11, srv1647850907=25, srv2044968456=52, srv1006678645=0, srv264643384=63, srv471583206=75, srv983031256=98, srv1042027882=2, srv1804691720=35, srv2044166373=51, srv1397098656=16, srv1973685291=45, srv2137320294=59, srv899605184=91, srv2034712193=49, srv1082862180=4, srv1670607822=26, srv886281159=90, srv2119463155=58, srv360987629=69, srv575756832=77, srv1055709457=3, srv607022969=79, srv1526983172=24, srv1791166257=33, srv644095809=84, srv1482270877=21, srv610226431=81, srv1322858487=14, srv1864352040=39, srv905883941=92, srv555399416=76, srv220631252=61, srv993492736=99, srv2000607841=46, srv1788900439=32, srv2081075314=54, srv711011958=85, srv1212429541=10, srv374689516=70, srv943233995=95, srv1451449700=20, srv19607250=43, srv2022084517=47, srv330135278=68, srv470921688=74, srv911858924=94, srv282981611=64, srv1322935552=15, srv579613773=78, srv713797429=86, srv410824557=73, srv2029601748=48, srv641680021=83, srv1786373837=30, srv908134147=93, srv962428461=96, srv1821749726=37, srv621282564=82, srv1296065405=13, srv1683177602=27, srv1168064955=8, srv283122155=65, srv1504144420=23, srv2119302110=57, srv798622233=87, srv1183749223=9, srv608958763=80, srv1879872708=40, srv1703831152=28, srv83002626=89, srv407220570=72, srv2096396523=56, srv970178054=97, srv1272564469=12, srv1796390744=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:11,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:11,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:11,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:11,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:11,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:11,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:11,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-02 11:53:11,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-02 11:53:11,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-02 11:53:11,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-02 11:53:11,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-02 11:53:11,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-02 11:53:11,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-02 11:53:11,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-02 11:53:11,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-02 11:53:11,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-02 11:53:11,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-02 11:53:12,132 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024073297060757656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03499238333028429, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024073297060757656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:12,132 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-02 11:53:12,132 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01584074175805355, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024073297060757656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03499238333028429, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024073297060757656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-02 11:53:12,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 627 ms to try 5000 different iterations. Found a solution that moves 950 regions; Going from a computed imbalance of 0.01584074175805355 to a new imbalance of 5.129038062317705E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006882042867432656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00843540185701534); MoveCostFunction : (multiplier=7.0, imbalance=0.10555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006882042867432656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:12,617 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1082862180:81 , srv2044166373:81 , srv316508715:82 , srv1010013152:83 , srv116316770:83 , srv1322935552:83 , srv150052848:83 , srv1967445334:83 , srv297330025:83 , srv1296065405:84 , srv1443539513:84 , srv1703831152:84 , srv1786373837:84 , srv1805431338:84 , srv2096396523:84 , srv2145937396:84 , srv407220570:84 , srv993492736:84 , srv1164830434:85 , srv1168064955:85 , srv1397098656:85 , srv1424365943:85 , srv1504144420:85 , srv1526983172:85 , srv1670607822:85 , srv1683177602:85 , srv1791166257:85 , srv1796390744:85 , srv1821749726:85 , srv1944372834:85 , srv1973685291:85 , srv2000607841:85 , srv2022084517:85 , srv2029601748:85 , srv203818760:85 , srv2044968456:85 , srv2067934561:85 , srv2092877441:85 , srv2137320294:85 , srv264643384:85 , srv282981611:85 , srv403359514:85 , srv470921688:85 , srv579613773:85 , srv607022969:85 , srv621282564:85 , srv711011958:85 , srv911858924:85 , srv943233995:85 , srv1006678645:86 , srv1143925668:86 , srv1183749223:86 , srv1322858487:86 , srv1427689385:86 , srv1451449700:86 , srv1647850907:86 , srv1744516773:86 , srv178739509:86 , srv1804691720:86 , srv1823342454:86 , srv1864352040:86 , srv1941372948:86 , srv19607250:86 , srv2119463155:86 , srv330135278:86 , srv374689516:86 , srv410824557:86 , srv471583206:86 , srv575756832:86 , srv610226431:86 , srv798622233:86 , srv83002626:86 , srv886281159:86 , srv899605184:86 , srv905883941:86 , srv908134147:86 , srv1042027882:87 , srv1055709457:87 , srv1212429541:87 , srv1272564469:87 , srv1482270877:87 , srv1879872708:87 , srv2081075314:87 , srv2119302110:87 , srv220631252:87 , srv360987629:87 , srv608958763:87 , srv644095809:87 , srv713797429:87 , srv983031256:87 , srv1263447659:88 , srv1788900439:88 , srv2034712193:88 , srv253289520:88 , srv555399416:88 , srv641680021:88 , srv82939676:88 , srv970178054:89 , srv283122155:91 , srv962428461:531 } 2023-08-02 11:53:12,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:12,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv253289520=62, srv316508715=67, srv150052848=22, srv403359514=71, srv1967445334=44, srv1010013152=1, srv1424365943=17, srv1143925668=5, srv1823342454=38, srv203818760=50, srv116316770=6, srv178739509=31, srv82939676=88, srv1427689385=18, srv1941372948=41, srv2145937396=60, srv1744516773=29, srv1164830434=7, srv1443539513=19, srv2067934561=53, srv1944372834=42, srv2092877441=55, srv297330025=66, srv1805431338=36, srv1263447659=11, srv1647850907=25, srv2044968456=52, srv1006678645=0, srv264643384=63, srv471583206=75, srv983031256=98, srv1042027882=2, srv1804691720=35, srv2044166373=51, srv1397098656=16, srv1973685291=45, srv2137320294=59, srv899605184=91, srv2034712193=49, srv1082862180=4, srv1670607822=26, srv886281159=90, srv2119463155=58, srv360987629=69, srv575756832=77, srv1055709457=3, srv607022969=79, srv1526983172=24, srv1791166257=33, srv644095809=84, srv1482270877=21, srv610226431=81, srv1322858487=14, srv1864352040=39, srv905883941=92, srv555399416=76, srv220631252=61, srv993492736=99, srv2000607841=46, srv1788900439=32, srv2081075314=54, srv711011958=85, srv1212429541=10, srv374689516=70, srv943233995=95, srv1451449700=20, srv19607250=43, srv2022084517=47, srv330135278=68, srv470921688=74, srv911858924=94, srv282981611=64, srv1322935552=15, srv579613773=78, srv713797429=86, srv410824557=73, srv2029601748=48, srv641680021=83, srv1786373837=30, srv908134147=93, srv962428461=96, srv1821749726=37, srv621282564=82, srv1296065405=13, srv1683177602=27, srv1168064955=8, srv283122155=65, srv1504144420=23, srv2119302110=57, srv798622233=87, srv1183749223=9, srv608958763=80, srv1879872708=40, srv1703831152=28, srv83002626=89, srv407220570=72, srv2096396523=56, srv970178054=97, srv1272564469=12, srv1796390744=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:12,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:12,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:12,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:12,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:12,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:12,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:12,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:12,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:12,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:12,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:12,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:12,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:12,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:12,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:12,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:12,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:12,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:12,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:12,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:12,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:12,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:12,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-02 11:53:12,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-02 11:53:12,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-02 11:53:12,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-02 11:53:12,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-02 11:53:12,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-02 11:53:12,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-02 11:53:12,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-02 11:53:12,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-02 11:53:12,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-02 11:53:12,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-02 11:53:12,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-02 11:53:12,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-02 11:53:12,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-02 11:53:12,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-02 11:53:12,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-02 11:53:12,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-02 11:53:12,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-02 11:53:12,833 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006882042867432656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00843540185701534); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006882042867432656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:12,833 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-02 11:53:12,836 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.062517640548221E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006882042867432656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00843540185701534); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006882042867432656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-02 11:53:13,205 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations. Found a solution that moves 275 regions; Going from a computed imbalance of 5.062517640548221E-4 to a new imbalance of 3.511458975793481E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.030555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:13,206 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1322858487:83 , srv1967445334:83 , srv283122155:83 , srv970178054:83 , srv1082862180:84 , srv1168064955:84 , srv1183749223:84 , srv1212429541:84 , srv1263447659:84 , srv1427689385:84 , srv1443539513:84 , srv1451449700:84 , srv150052848:84 , srv1647850907:84 , srv1683177602:84 , srv1703831152:84 , srv1744516773:84 , srv178739509:84 , srv1796390744:84 , srv1821749726:84 , srv1823342454:84 , srv1944372834:84 , srv19607250:84 , srv2067934561:84 , srv2081075314:84 , srv2092877441:84 , srv2096396523:84 , srv2119302110:84 , srv2119463155:84 , srv2145937396:84 , srv330135278:84 , srv374689516:84 , srv471583206:84 , srv555399416:84 , srv607022969:84 , srv641680021:84 , srv711011958:84 , srv713797429:84 , srv82939676:84 , srv83002626:84 , srv905883941:84 , srv908134147:84 , srv911858924:84 , srv983031256:84 , srv1006678645:85 , srv1010013152:85 , srv1042027882:85 , srv1143925668:85 , srv116316770:85 , srv1164830434:85 , srv1272564469:85 , srv1296065405:85 , srv1322935552:85 , srv1397098656:85 , srv1424365943:85 , srv1482270877:85 , srv1504144420:85 , srv1526983172:85 , srv1670607822:85 , srv1788900439:85 , srv1791166257:85 , srv1805431338:85 , srv1864352040:85 , srv1879872708:85 , srv1973685291:85 , srv2000607841:85 , srv2029601748:85 , srv203818760:85 , srv2044968456:85 , srv2137320294:85 , srv220631252:85 , srv253289520:85 , srv264643384:85 , srv282981611:85 , srv297330025:85 , srv316508715:85 , srv407220570:85 , srv410824557:85 , srv470921688:85 , srv575756832:85 , srv579613773:85 , srv608958763:85 , srv621282564:85 , srv644095809:85 , srv798622233:85 , srv886281159:85 , srv899605184:85 , srv943233995:85 , srv993492736:85 , srv1055709457:86 , srv1786373837:86 , srv1804691720:86 , srv1941372948:86 , srv2022084517:86 , srv2034712193:86 , srv2044166373:86 , srv360987629:86 , srv403359514:86 , srv610226431:86 , srv962428461:623 } 2023-08-02 11:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv253289520=62, srv316508715=67, srv150052848=22, srv403359514=71, srv1967445334=44, srv1010013152=1, srv1424365943=17, srv1143925668=5, srv1823342454=38, srv203818760=50, srv116316770=6, srv178739509=31, srv82939676=88, srv1427689385=18, srv1941372948=41, srv2145937396=60, srv1744516773=29, srv1164830434=7, srv1443539513=19, srv2067934561=53, srv1944372834=42, srv2092877441=55, srv297330025=66, srv1805431338=36, srv1263447659=11, srv1647850907=25, srv2044968456=52, srv1006678645=0, srv264643384=63, srv471583206=75, srv983031256=98, srv1042027882=2, srv1804691720=35, srv2044166373=51, srv1397098656=16, srv1973685291=45, srv2137320294=59, srv899605184=91, srv2034712193=49, srv1082862180=4, srv1670607822=26, srv886281159=90, srv2119463155=58, srv360987629=69, srv575756832=77, srv1055709457=3, srv607022969=79, srv1526983172=24, srv1791166257=33, srv644095809=84, srv1482270877=21, srv610226431=81, srv1322858487=14, srv1864352040=39, srv905883941=92, srv555399416=76, srv220631252=61, srv993492736=99, srv2000607841=46, srv1788900439=32, srv2081075314=54, srv711011958=85, srv1212429541=10, srv374689516=70, srv943233995=95, srv1451449700=20, srv19607250=43, srv2022084517=47, srv330135278=68, srv470921688=74, srv911858924=94, srv282981611=64, srv1322935552=15, srv579613773=78, srv713797429=86, srv410824557=73, srv2029601748=48, srv641680021=83, srv1786373837=30, srv908134147=93, srv962428461=96, srv1821749726=37, srv621282564=82, srv1296065405=13, srv1683177602=27, srv1168064955=8, srv283122155=65, srv1504144420=23, srv2119302110=57, srv798622233=87, srv1183749223=9, srv608958763=80, srv1879872708=40, srv1703831152=28, srv83002626=89, srv407220570=72, srv2096396523=56, srv970178054=97, srv1272564469=12, srv1796390744=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-02 11:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-02 11:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-02 11:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-02 11:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-02 11:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-02 11:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-02 11:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-02 11:53:13,364 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:13,364 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-02 11:53:13,365 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.3188998601449776E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-02 11:53:13,704 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations. Found a solution that moves 22 regions; Going from a computed imbalance of 3.3188998601449776E-5 to a new imbalance of 7.541804792831362E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0024444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:13,704 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1143925668:83 , srv1941372948:83 , srv1973685291:83 , srv2034712193:83 , srv360987629:83 , srv403359514:83 , srv410824557:83 , srv610226431:83 , srv1006678645:84 , srv1010013152:84 , srv1042027882:84 , srv1055709457:84 , srv116316770:84 , srv1164830434:84 , srv1168064955:84 , srv1183749223:84 , srv1212429541:84 , srv1263447659:84 , srv1272564469:84 , srv1296065405:84 , srv1322935552:84 , srv1397098656:84 , srv1424365943:84 , srv1427689385:84 , srv1443539513:84 , srv1482270877:84 , srv1504144420:84 , srv1526983172:84 , srv1647850907:84 , srv1670607822:84 , srv1683177602:84 , srv1744516773:84 , srv1786373837:84 , srv178739509:84 , srv1788900439:84 , srv1791166257:84 , srv1796390744:84 , srv1804691720:84 , srv1805431338:84 , srv1821749726:84 , srv1823342454:84 , srv1864352040:84 , srv1879872708:84 , srv19607250:84 , srv1967445334:84 , srv2000607841:84 , srv2022084517:84 , srv2029601748:84 , srv203818760:84 , srv2044166373:84 , srv2044968456:84 , srv2067934561:84 , srv2081075314:84 , srv2092877441:84 , srv2096396523:84 , srv2119302110:84 , srv2119463155:84 , srv2137320294:84 , srv220631252:84 , srv253289520:84 , srv264643384:84 , srv282981611:84 , srv283122155:84 , srv297330025:84 , srv316508715:84 , srv330135278:84 , srv374689516:84 , srv407220570:84 , srv470921688:84 , srv471583206:84 , srv555399416:84 , srv575756832:84 , srv579613773:84 , srv607022969:84 , srv608958763:84 , srv621282564:84 , srv641680021:84 , srv644095809:84 , srv711011958:84 , srv713797429:84 , srv798622233:84 , srv82939676:84 , srv83002626:84 , srv886281159:84 , srv899605184:84 , srv905883941:84 , srv908134147:84 , srv943233995:84 , srv970178054:84 , srv983031256:84 , srv993492736:84 , srv1082862180:85 , srv1322858487:85 , srv1451449700:85 , srv150052848:85 , srv1703831152:85 , srv1944372834:85 , srv2145937396:85 , srv911858924:85 , srv962428461:684 } 2023-08-02 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv253289520=62, srv316508715=67, srv150052848=22, srv403359514=71, srv1967445334=44, srv1010013152=1, srv1424365943=17, srv1143925668=5, srv1823342454=38, srv203818760=50, srv116316770=6, srv178739509=31, srv82939676=88, srv1427689385=18, srv1941372948=41, srv2145937396=60, srv1744516773=29, srv1164830434=7, srv1443539513=19, srv2067934561=53, srv1944372834=42, srv2092877441=55, srv297330025=66, srv1805431338=36, srv1263447659=11, srv1647850907=25, srv2044968456=52, srv1006678645=0, srv264643384=63, srv471583206=75, srv983031256=98, srv1042027882=2, srv1804691720=35, srv2044166373=51, srv1397098656=16, srv1973685291=45, srv2137320294=59, srv899605184=91, srv2034712193=49, srv1082862180=4, srv1670607822=26, srv886281159=90, srv2119463155=58, srv360987629=69, srv575756832=77, srv1055709457=3, srv607022969=79, srv1526983172=24, srv1791166257=33, srv644095809=84, srv1482270877=21, srv610226431=81, srv1322858487=14, srv1864352040=39, srv905883941=92, srv555399416=76, srv220631252=61, srv993492736=99, srv2000607841=46, srv1788900439=32, srv2081075314=54, srv711011958=85, srv1212429541=10, srv374689516=70, srv943233995=95, srv1451449700=20, srv19607250=43, srv2022084517=47, srv330135278=68, srv470921688=74, srv911858924=94, srv282981611=64, srv1322935552=15, srv579613773=78, srv713797429=86, srv410824557=73, srv2029601748=48, srv641680021=83, srv1786373837=30, srv908134147=93, srv962428461=96, srv1821749726=37, srv621282564=82, srv1296065405=13, srv1683177602=27, srv1168064955=8, srv283122155=65, srv1504144420=23, srv2119302110=57, srv798622233=87, srv1183749223=9, srv608958763=80, srv1879872708=40, srv1703831152=28, srv83002626=89, srv407220570=72, srv2096396523=56, srv970178054=97, srv1272564469=12, srv1796390744=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-02 11:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-02 11:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-02 11:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-02 11:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-02 11:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-02 11:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-02 11:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-02 11:53:13,883 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:13,883 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1143925668:83 , srv1941372948:83 , srv1973685291:83 , srv2034712193:83 , srv360987629:83 , srv403359514:83 , srv410824557:83 , srv610226431:83 , srv1006678645:84 , srv1010013152:84 , srv1042027882:84 , srv1055709457:84 , srv116316770:84 , srv1164830434:84 , srv1168064955:84 , srv1183749223:84 , srv1212429541:84 , srv1263447659:84 , srv1272564469:84 , srv1296065405:84 , srv1322935552:84 , srv1397098656:84 , srv1424365943:84 , srv1427689385:84 , srv1443539513:84 , srv1482270877:84 , srv1504144420:84 , srv1526983172:84 , srv1647850907:84 , srv1670607822:84 , srv1683177602:84 , srv1744516773:84 , srv1786373837:84 , srv178739509:84 , srv1788900439:84 , srv1791166257:84 , srv1796390744:84 , srv1804691720:84 , srv1805431338:84 , srv1821749726:84 , srv1823342454:84 , srv1864352040:84 , srv1879872708:84 , srv19607250:84 , srv1967445334:84 , srv2000607841:84 , srv2022084517:84 , srv2029601748:84 , srv203818760:84 , srv2044166373:84 , srv2044968456:84 , srv2067934561:84 , srv2081075314:84 , srv2092877441:84 , srv2096396523:84 , srv2119302110:84 , srv2119463155:84 , srv2137320294:84 , srv220631252:84 , srv253289520:84 , srv264643384:84 , srv282981611:84 , srv283122155:84 , srv297330025:84 , srv316508715:84 , srv330135278:84 , srv374689516:84 , srv407220570:84 , srv470921688:84 , srv471583206:84 , srv555399416:84 , srv575756832:84 , srv579613773:84 , srv607022969:84 , srv608958763:84 , srv621282564:84 , srv641680021:84 , srv644095809:84 , srv711011958:84 , srv713797429:84 , srv798622233:84 , srv82939676:84 , srv83002626:84 , srv886281159:84 , srv899605184:84 , srv905883941:84 , srv908134147:84 , srv943233995:84 , srv970178054:84 , srv983031256:84 , srv993492736:84 , srv1082862180:85 , srv1322858487:85 , srv1451449700:85 , srv150052848:85 , srv1703831152:85 , srv1944372834:85 , srv2145937396:85 , srv911858924:85 , srv962428461:684 } 2023-08-02 11:53:13,912 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=332 (was 309) - SystemLoadAverage LEAK? -, ProcessCount=176 (was 174) - ProcessCount LEAK? -, AvailableMemoryMB=2045 (was 2922) 2023-08-02 11:53:13,923 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=332, ProcessCount=174, AvailableMemoryMB=2045 2023-08-02 11:53:13,924 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-02 11:53:13,924 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-02 11:53:13,936 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1033292758:84 , srv104579458:84 , srv1165157454:84 , srv1286288985:84 , srv1293665420:84 , srv1294584133:84 , srv1311786845:84 , srv1388965501:84 , srv1403565587:84 , srv1425725196:84 , srv1452095421:84 , srv1537904102:84 , srv1545676443:84 , srv1580098128:84 , srv1613909037:84 , srv1627799796:84 , srv1654645341:84 , srv1718055612:84 , srv1888261492:84 , srv2146354370:84 , srv376444751:84 , srv481840798:84 , srv524467720:84 , srv59609859:84 , srv642064757:84 , srv714309680:84 , srv745056137:84 , srv849969235:84 , srv859005513:84 , srv57711160:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-02 11:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1425725196=9, srv745056137=27, srv481840798=21, srv1293665420=4, srv1654645341=16, srv1613909037=14, srv1718055612=17, srv1452095421=10, srv1388965501=7, srv1545676443=12, srv1627799796=15, srv642064757=25, srv104579458=1, srv2146354370=19, srv376444751=20, srv59609859=24, srv1311786845=6, srv1294584133=5, srv1165157454=2, srv714309680=26, srv1033292758=0, srv1403565587=8, srv849969235=28, srv1888261492=18, srv1580098128=13, srv524467720=22, srv1286288985=3, srv57711160=23, srv859005513=29, srv1537904102=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:53:13,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:53:13,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:53:13,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-02 11:53:13,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1236623246251847, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:53:13,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912582699211228, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1236623246251847, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-02 11:55:13,960 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5368082 different iterations. Found a solution that moves 1863 regions; Going from a computed imbalance of 0.9912582699211228 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:55:13,966 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1033292758:90 , srv104579458:90 , srv1165157454:90 , srv1286288985:90 , srv1293665420:90 , srv1294584133:90 , srv1311786845:90 , srv1388965501:90 , srv1403565587:90 , srv1425725196:90 , srv1452095421:90 , srv1537904102:90 , srv1545676443:90 , srv1580098128:90 , srv1613909037:90 , srv1627799796:90 , srv1654645341:90 , srv1718055612:90 , srv1888261492:90 , srv2146354370:90 , srv376444751:90 , srv481840798:90 , srv524467720:90 , srv57711160:90 , srv59609859:90 , srv642064757:90 , srv714309680:90 , srv745056137:90 , srv849969235:90 , srv859005513:90 } 2023-08-02 11:55:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-02 11:55:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1425725196=9, srv745056137=27, srv481840798=21, srv1293665420=4, srv1654645341=16, srv1613909037=14, srv1718055612=17, srv1452095421=10, srv1388965501=7, srv1545676443=12, srv1627799796=15, srv642064757=25, srv104579458=1, srv2146354370=19, srv376444751=20, srv59609859=24, srv1311786845=6, srv1294584133=5, srv1165157454=2, srv714309680=26, srv1033292758=0, srv1403565587=8, srv849969235=28, srv1888261492=18, srv1580098128=13, srv524467720=22, srv1286288985=3, srv57711160=23, srv859005513=29, srv1537904102=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-02 11:55:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-02 11:55:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-02 11:55:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-02 11:55:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-02 11:55:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-02 11:55:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-02 11:55:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-02 11:55:13,984 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-02 11:55:13,984 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1033292758:90 , srv104579458:90 , srv1165157454:90 , srv1286288985:90 , srv1293665420:90 , srv1294584133:90 , srv1311786845:90 , srv1388965501:90 , srv1403565587:90 , srv1425725196:90 , srv1452095421:90 , srv1537904102:90 , srv1545676443:90 , srv1580098128:90 , srv1613909037:90 , srv1627799796:90 , srv1654645341:90 , srv1718055612:90 , srv1888261492:90 , srv2146354370:90 , srv376444751:90 , srv481840798:90 , srv524467720:90 , srv57711160:90 , srv59609859:90 , srv642064757:90 , srv714309680:90 , srv745056137:90 , srv849969235:90 , srv859005513:90 } 2023-08-02 11:55:13,999 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=192 (was 332), ProcessCount=171 (was 174), AvailableMemoryMB=1930 (was 2045)