2023-10-09 06:53:27,397 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-09 06:53:27,687 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-09 06:53:27,805 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-09 06:53:27,874 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-09 06:53:27,882 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-09 06:53:27,897 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=170, AvailableMemoryMB=9173 2023-10-09 06:53:27,899 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-09 06:53:27,900 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-09 06:53:28,038 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003842088:84 , srv1010713630:84 , srv1017578764:84 , srv1029165728:84 , srv1056165619:84 , srv1061392174:84 , srv1092366914:84 , srv1117176008:84 , srv1125560578:84 , srv1139136941:84 , srv1142585544:84 , srv1157683362:84 , srv1172059746:84 , srv1192272112:84 , srv1194200830:84 , srv1197528851:84 , srv1207062693:84 , srv1243092650:84 , srv1281383465:84 , srv1296833299:84 , srv1366950860:84 , srv1392776121:84 , srv1422612740:84 , srv1433809142:84 , srv1456191102:84 , srv146074749:84 , srv1462261701:84 , srv149080593:84 , srv1496601302:84 , srv1510361780:84 , srv1570578810:84 , srv1576879099:84 , srv1594313192:84 , srv159862055:84 , srv161827955:84 , srv1667673256:84 , srv1681862375:84 , srv1684425784:84 , srv1694005180:84 , srv1734811335:84 , srv1821445845:84 , srv184683603:84 , srv1857728177:84 , srv1870338922:84 , srv1885545551:84 , srv1901503834:84 , srv1901856678:84 , srv1918889029:84 , srv1937075575:84 , srv1951787487:84 , srv1976207523:84 , srv1976311830:84 , srv1977312673:84 , srv1995773489:84 , srv2076718082:84 , srv2084820580:84 , srv2086374528:84 , srv2109889199:84 , srv211014239:84 , srv2131072036:84 , srv2147017256:84 , srv218358243:84 , srv239780194:84 , srv243066495:84 , srv283521566:84 , srv378025476:84 , srv405049219:84 , srv415252475:84 , srv482791625:84 , srv498037065:84 , srv515122492:84 , srv52890093:84 , srv557774186:84 , srv559943641:84 , srv563080538:84 , srv579224863:84 , srv580638772:84 , srv592455550:84 , srv606463801:84 , srv6563482:84 , srv658225910:84 , srv66971886:84 , srv699217932:84 , srv706075030:84 , srv72193591:84 , srv756991529:84 , srv775209016:84 , srv776057433:84 , srv788008886:84 , srv78867977:84 , srv795147856:84 , srv823208837:84 , srv867705526:84 , srv910327457:84 , srv920216613:84 , srv947733639:84 , srv952944997:84 , srv967971771:84 , srv981360614:84 , srv410832460:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-09 06:53:28,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:53:28,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1192272112=13, srv218358243=61, srv1281383465=18, srv159862055=33, srv161827955=34, srv78867977=90, srv146074749=25, srv1918889029=47, srv1995773489=53, srv1003842088=0, srv482791625=69, srv1433809142=23, srv1296833299=19, srv1010713630=1, srv823208837=92, srv1462261701=26, srv1901856678=46, srv52890093=72, srv592455550=78, srv2131072036=59, srv1870338922=43, srv239780194=62, srv1142585544=10, srv1056165619=4, srv1172059746=12, srv1857728177=42, srv580638772=77, srv2147017256=60, srv1510361780=29, srv1885545551=44, srv1976311830=51, srv405049219=66, srv6563482=80, srv776057433=88, srv66971886=82, srv563080538=75, srv1029165728=3, srv2086374528=56, srv867705526=93, srv378025476=65, srv515122492=71, srv498037065=70, srv706075030=84, srv1901503834=45, srv1734811335=39, srv788008886=89, srv1821445845=40, srv149080593=27, srv756991529=86, srv1422612740=22, srv1243092650=17, srv1976207523=50, srv952944997=97, srv775209016=87, srv579224863=76, srv920216613=95, srv1061392174=5, srv1570578810=30, srv1125560578=8, srv184683603=41, srv1684425784=37, srv658225910=81, srv283521566=64, srv72193591=85, srv211014239=58, srv1951787487=49, srv1366950860=20, srv1681862375=36, srv1157683362=11, srv1694005180=38, srv1194200830=14, srv1117176008=7, srv795147856=91, srv559943641=74, srv1937075575=48, srv1092366914=6, srv1197528851=15, srv981360614=99, srv1667673256=35, srv2076718082=54, srv243066495=63, srv606463801=79, srv699217932=83, srv1576879099=31, srv2084820580=55, srv967971771=98, srv1456191102=24, srv415252475=68, srv1594313192=32, srv557774186=73, srv1392776121=21, srv1977312673=52, srv410832460=67, srv1139136941=9, srv1207062693=16, srv2109889199=57, srv947733639=96, srv1017578764=2, srv1496601302=28, srv910327457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:53:28,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:53:28,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:53:28,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:53:28,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:53:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-09 06:53:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-09 06:53:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-09 06:53:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-09 06:53:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-09 06:53:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-09 06:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-09 06:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-09 06:53:28,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:28,335 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-09 06:53:28,336 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-09 06:53:28,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 540 ms to try 5000 different iterations. Found a solution that moves 4685 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1794745971324284. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03758815551595551, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05682588051097284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5205555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03758815551595551, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:28,728 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1901503834:81 , srv1684425784:82 , srv415252475:82 , srv1017578764:83 , srv1510361780:83 , srv2147017256:83 , srv1010713630:84 , srv1422612740:84 , srv1456191102:84 , srv149080593:84 , srv1734811335:84 , srv1995773489:84 , srv2084820580:84 , srv2131072036:84 , srv378025476:84 , srv482791625:84 , srv52890093:84 , srv579224863:84 , srv580638772:84 , srv72193591:84 , srv756991529:84 , srv867705526:84 , srv920216613:84 , srv1142585544:85 , srv1192272112:85 , srv1243092650:85 , srv1296833299:85 , srv1821445845:85 , srv1870338922:85 , srv1901856678:85 , srv218358243:85 , srv239780194:85 , srv243066495:85 , srv283521566:85 , srv6563482:85 , srv66971886:85 , srv1139136941:86 , srv1157683362:86 , srv1281383465:86 , srv146074749:86 , srv1594313192:86 , srv1667673256:86 , srv1977312673:86 , srv2109889199:86 , srv405049219:86 , srv498037065:86 , srv515122492:86 , srv823208837:86 , srv1003842088:87 , srv1029165728:87 , srv1172059746:87 , srv1197528851:87 , srv1570578810:87 , srv159862055:87 , srv1694005180:87 , srv1857728177:87 , srv1885545551:87 , srv2076718082:87 , srv211014239:87 , srv557774186:87 , srv592455550:87 , srv658225910:87 , srv706075030:87 , srv947733639:87 , srv952944997:87 , srv981360614:87 , srv161827955:88 , srv1937075575:88 , srv1951787487:88 , srv1976207523:88 , srv1976311830:88 , srv559943641:88 , srv606463801:88 , srv699217932:88 , srv775209016:88 , srv776057433:88 , srv967971771:88 , srv1056165619:89 , srv1061392174:89 , srv1125560578:89 , srv1194200830:89 , srv1392776121:89 , srv1576879099:89 , srv1681862375:89 , srv184683603:89 , srv2086374528:89 , srv563080538:89 , srv795147856:89 , srv1433809142:90 , srv1462261701:90 , srv1496601302:90 , srv78867977:90 , srv1092366914:91 , srv1918889029:91 , srv788008886:91 , srv1207062693:92 , srv1366950860:92 , srv910327457:92 , srv1117176008:95 , srv410832460:424 } 2023-10-09 06:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1192272112=13, srv218358243=61, srv1281383465=18, srv159862055=33, srv161827955=34, srv78867977=90, srv146074749=25, srv1918889029=47, srv1995773489=53, srv1003842088=0, srv482791625=69, srv1433809142=23, srv1296833299=19, srv1010713630=1, srv823208837=92, srv1462261701=26, srv1901856678=46, srv52890093=72, srv592455550=78, srv2131072036=59, srv1870338922=43, srv239780194=62, srv1142585544=10, srv1056165619=4, srv1172059746=12, srv1857728177=42, srv580638772=77, srv2147017256=60, srv1510361780=29, srv1885545551=44, srv1976311830=51, srv405049219=66, srv6563482=80, srv776057433=88, srv66971886=82, srv563080538=75, srv1029165728=3, srv2086374528=56, srv867705526=93, srv378025476=65, srv515122492=71, srv498037065=70, srv706075030=84, srv1901503834=45, srv1734811335=39, srv788008886=89, srv1821445845=40, srv149080593=27, srv756991529=86, srv1422612740=22, srv1243092650=17, srv1976207523=50, srv952944997=97, srv775209016=87, srv579224863=76, srv920216613=95, srv1061392174=5, srv1570578810=30, srv1125560578=8, srv184683603=41, srv1684425784=37, srv658225910=81, srv283521566=64, srv72193591=85, srv211014239=58, srv1951787487=49, srv1366950860=20, srv1681862375=36, srv1157683362=11, srv1694005180=38, srv1194200830=14, srv1117176008=7, srv795147856=91, srv559943641=74, srv1937075575=48, srv1092366914=6, srv1197528851=15, srv981360614=99, srv1667673256=35, srv2076718082=54, srv243066495=63, srv606463801=79, srv699217932=83, srv1576879099=31, srv2084820580=55, srv967971771=98, srv1456191102=24, srv415252475=68, srv1594313192=32, srv557774186=73, srv1392776121=21, srv1977312673=52, srv410832460=67, srv1139136941=9, srv1207062693=16, srv2109889199=57, srv947733639=96, srv1017578764=2, srv1496601302=28, srv910327457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:53:28,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:53:28,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:53:28,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:53:28,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:53:28,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:53:28,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-09 06:53:28,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-09 06:53:28,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-09 06:53:28,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-09 06:53:28,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-09 06:53:28,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-09 06:53:28,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-09 06:53:28,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-09 06:53:28,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-09 06:53:28,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-09 06:53:28,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03758815551595551, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05682588051097284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03758815551595551, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:28,937 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-09 06:53:28,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17944179206127156, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03758815551595551, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05682588051097284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03758815551595551, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-09 06:53:29,246 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations. Found a solution that moves 2661 regions; Going from a computed imbalance of 0.17944179206127156 to a new imbalance of 0.015131375741311496. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02126912334655952); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036265017787828466, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2956666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02126912334655952); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:29,260 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1117176008:76 , srv1207062693:77 , srv1092366914:79 , srv1918889029:79 , srv910327457:80 , srv1194200830:81 , srv1366950860:81 , srv1462261701:81 , srv1681862375:81 , srv1056165619:82 , srv1125560578:82 , srv795147856:82 , srv1061392174:83 , srv1392776121:83 , srv1433809142:83 , srv1496601302:83 , srv1576879099:83 , srv1694005180:83 , srv184683603:83 , srv1976207523:83 , srv559943641:83 , srv563080538:83 , srv78867977:83 , srv1029165728:84 , srv1157683362:84 , srv1197528851:84 , srv1281383465:84 , srv161827955:84 , srv1976311830:84 , srv2076718082:84 , srv592455550:84 , srv606463801:84 , srv699217932:84 , srv775209016:84 , srv788008886:84 , srv947733639:84 , srv1010713630:85 , srv1192272112:85 , srv159862055:85 , srv1857728177:85 , srv1885545551:85 , srv2086374528:85 , srv405049219:85 , srv498037065:85 , srv579224863:85 , srv776057433:85 , srv823208837:85 , srv952944997:85 , srv981360614:85 , srv1139136941:86 , srv1570578810:86 , srv1937075575:86 , srv239780194:86 , srv283521566:86 , srv557774186:86 , srv658225910:86 , srv706075030:86 , srv967971771:86 , srv1003842088:87 , srv1142585544:87 , srv1172059746:87 , srv1296833299:87 , srv1734811335:87 , srv1870338922:87 , srv1901856678:87 , srv2084820580:87 , srv211014239:87 , srv2131072036:87 , srv6563482:87 , srv756991529:87 , srv1017578764:88 , srv1243092650:88 , srv1422612740:88 , srv146074749:88 , srv149080593:88 , srv1667673256:88 , srv1821445845:88 , srv1951787487:88 , srv1977312673:88 , srv2109889199:88 , srv2147017256:88 , srv218358243:88 , srv243066495:88 , srv378025476:88 , srv515122492:88 , srv52890093:88 , srv66971886:88 , srv1456191102:89 , srv1510361780:89 , srv1594313192:89 , srv1684425784:89 , srv1995773489:89 , srv580638772:89 , srv415252475:90 , srv482791625:90 , srv72193591:90 , srv867705526:90 , srv920216613:90 , srv1901503834:92 , srv410832460:539 } 2023-10-09 06:53:29,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:53:29,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1192272112=13, srv218358243=61, srv1281383465=18, srv159862055=33, srv161827955=34, srv78867977=90, srv146074749=25, srv1918889029=47, srv1995773489=53, srv1003842088=0, srv482791625=69, srv1433809142=23, srv1296833299=19, srv1010713630=1, srv823208837=92, srv1462261701=26, srv1901856678=46, srv52890093=72, srv592455550=78, srv2131072036=59, srv1870338922=43, srv239780194=62, srv1142585544=10, srv1056165619=4, srv1172059746=12, srv1857728177=42, srv580638772=77, srv2147017256=60, srv1510361780=29, srv1885545551=44, srv1976311830=51, srv405049219=66, srv6563482=80, srv776057433=88, srv66971886=82, srv563080538=75, srv1029165728=3, srv2086374528=56, srv867705526=93, srv378025476=65, srv515122492=71, srv498037065=70, srv706075030=84, srv1901503834=45, srv1734811335=39, srv788008886=89, srv1821445845=40, srv149080593=27, srv756991529=86, srv1422612740=22, srv1243092650=17, srv1976207523=50, srv952944997=97, srv775209016=87, srv579224863=76, srv920216613=95, srv1061392174=5, srv1570578810=30, srv1125560578=8, srv184683603=41, srv1684425784=37, srv658225910=81, srv283521566=64, srv72193591=85, srv211014239=58, srv1951787487=49, srv1366950860=20, srv1681862375=36, srv1157683362=11, srv1694005180=38, srv1194200830=14, srv1117176008=7, srv795147856=91, srv559943641=74, srv1937075575=48, srv1092366914=6, srv1197528851=15, srv981360614=99, srv1667673256=35, srv2076718082=54, srv243066495=63, srv606463801=79, srv699217932=83, srv1576879099=31, srv2084820580=55, srv967971771=98, srv1456191102=24, srv415252475=68, srv1594313192=32, srv557774186=73, srv1392776121=21, srv1977312673=52, srv410832460=67, srv1139136941=9, srv1207062693=16, srv2109889199=57, srv947733639=96, srv1017578764=2, srv1496601302=28, srv910327457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:53:29,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:53:29,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-09 06:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-09 06:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-09 06:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-09 06:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-09 06:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-09 06:53:29,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-09 06:53:29,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-09 06:53:29,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-09 06:53:29,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-09 06:53:29,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-09 06:53:29,411 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02126912334655952); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036265017787828466, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02126912334655952); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:29,411 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-09 06:53:29,411 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01511274302106638, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02126912334655952); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036265017787828466, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02126912334655952); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-09 06:53:29,679 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations. Found a solution that moves 1056 regions; Going from a computed imbalance of 0.01511274302106638 to a new imbalance of 5.186539627561537E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00365626180754161); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004666185641394383); MoveCostFunction : (multiplier=7.0, imbalance=0.11733333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00365626180754161); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:29,681 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1951787487:81 , srv1594313192:82 , srv1937075575:83 , srv2086374528:83 , srv211014239:83 , srv72193591:83 , srv788008886:83 , srv867705526:83 , srv1003842088:84 , srv1172059746:84 , srv1456191102:84 , srv146074749:84 , srv1667673256:84 , srv1977312673:84 , srv1995773489:84 , srv2109889199:84 , srv243066495:84 , srv482791625:84 , srv515122492:84 , srv557774186:84 , srv580638772:84 , srv78867977:84 , srv920216613:84 , srv1142585544:85 , srv1243092650:85 , srv1296833299:85 , srv1366950860:85 , srv1422612740:85 , srv1433809142:85 , srv1496601302:85 , srv1570578810:85 , srv1821445845:85 , srv184683603:85 , srv1857728177:85 , srv1870338922:85 , srv1901503834:85 , srv1901856678:85 , srv1976311830:85 , srv415252475:85 , srv52890093:85 , srv606463801:85 , srv66971886:85 , srv699217932:85 , srv706075030:85 , srv776057433:85 , srv967971771:85 , srv1029165728:86 , srv1056165619:86 , srv1061392174:86 , srv1117176008:86 , srv1139136941:86 , srv1392776121:86 , srv149080593:86 , srv1576879099:86 , srv159862055:86 , srv1885545551:86 , srv1976207523:86 , srv2084820580:86 , srv378025476:86 , srv498037065:86 , srv563080538:86 , srv592455550:86 , srv6563482:86 , srv658225910:86 , srv775209016:86 , srv795147856:86 , srv823208837:86 , srv910327457:86 , srv952944997:86 , srv981360614:86 , srv1017578764:87 , srv1092366914:87 , srv1125560578:87 , srv1157683362:87 , srv1194200830:87 , srv1197528851:87 , srv1281383465:87 , srv1462261701:87 , srv1510361780:87 , srv161827955:87 , srv1684425784:87 , srv1734811335:87 , srv2076718082:87 , srv2131072036:87 , srv2147017256:87 , srv218358243:87 , srv239780194:87 , srv283521566:87 , srv405049219:87 , srv559943641:87 , srv756991529:87 , srv947733639:87 , srv1192272112:88 , srv1681862375:88 , srv1918889029:88 , srv579224863:88 , srv1010713630:89 , srv1207062693:89 , srv1694005180:89 , srv410832460:527 } 2023-10-09 06:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1192272112=13, srv218358243=61, srv1281383465=18, srv159862055=33, srv161827955=34, srv78867977=90, srv146074749=25, srv1918889029=47, srv1995773489=53, srv1003842088=0, srv482791625=69, srv1433809142=23, srv1296833299=19, srv1010713630=1, srv823208837=92, srv1462261701=26, srv1901856678=46, srv52890093=72, srv592455550=78, srv2131072036=59, srv1870338922=43, srv239780194=62, srv1142585544=10, srv1056165619=4, srv1172059746=12, srv1857728177=42, srv580638772=77, srv2147017256=60, srv1510361780=29, srv1885545551=44, srv1976311830=51, srv405049219=66, srv6563482=80, srv776057433=88, srv66971886=82, srv563080538=75, srv1029165728=3, srv2086374528=56, srv867705526=93, srv378025476=65, srv515122492=71, srv498037065=70, srv706075030=84, srv1901503834=45, srv1734811335=39, srv788008886=89, srv1821445845=40, srv149080593=27, srv756991529=86, srv1422612740=22, srv1243092650=17, srv1976207523=50, srv952944997=97, srv775209016=87, srv579224863=76, srv920216613=95, srv1061392174=5, srv1570578810=30, srv1125560578=8, srv184683603=41, srv1684425784=37, srv658225910=81, srv283521566=64, srv72193591=85, srv211014239=58, srv1951787487=49, srv1366950860=20, srv1681862375=36, srv1157683362=11, srv1694005180=38, srv1194200830=14, srv1117176008=7, srv795147856=91, srv559943641=74, srv1937075575=48, srv1092366914=6, srv1197528851=15, srv981360614=99, srv1667673256=35, srv2076718082=54, srv243066495=63, srv606463801=79, srv699217932=83, srv1576879099=31, srv2084820580=55, srv967971771=98, srv1456191102=24, srv415252475=68, srv1594313192=32, srv557774186=73, srv1392776121=21, srv1977312673=52, srv410832460=67, srv1139136941=9, srv1207062693=16, srv2109889199=57, srv947733639=96, srv1017578764=2, srv1496601302=28, srv910327457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-09 06:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-09 06:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-09 06:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-09 06:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-09 06:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-09 06:53:29,831 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00365626180754161); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004666185641394383); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00365626180754161); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:29,831 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-09 06:53:29,832 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.112596927152512E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00365626180754161); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004666185641394383); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00365626180754161); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-09 06:53:30,303 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 584 ms to try 5000 different iterations. Found a solution that moves 192 regions; Going from a computed imbalance of 5.112596927152512E-4 to a new imbalance of 1.852726984150397E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.021333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:30,304 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv218358243:82 , srv1197528851:83 , srv1510361780:83 , srv161827955:83 , srv1694005180:83 , srv658225910:83 , srv967971771:83 , srv1003842088:84 , srv1010713630:84 , srv1017578764:84 , srv1029165728:84 , srv1061392174:84 , srv1139136941:84 , srv1172059746:84 , srv1192272112:84 , srv1207062693:84 , srv1243092650:84 , srv1366950860:84 , srv1392776121:84 , srv1422612740:84 , srv1433809142:84 , srv146074749:84 , srv1462261701:84 , srv149080593:84 , srv1496601302:84 , srv1570578810:84 , srv1576879099:84 , srv159862055:84 , srv1667673256:84 , srv1681862375:84 , srv1684425784:84 , srv1734811335:84 , srv1870338922:84 , srv1885545551:84 , srv1901503834:84 , srv1918889029:84 , srv1977312673:84 , srv2076718082:84 , srv2084820580:84 , srv2131072036:84 , srv2147017256:84 , srv239780194:84 , srv283521566:84 , srv378025476:84 , srv405049219:84 , srv515122492:84 , srv559943641:84 , srv563080538:84 , srv6563482:84 , srv66971886:84 , srv706075030:84 , srv775209016:84 , srv776057433:84 , srv788008886:84 , srv910327457:84 , srv920216613:84 , srv947733639:84 , srv952944997:84 , srv981360614:84 , srv1056165619:85 , srv1092366914:85 , srv1117176008:85 , srv1125560578:85 , srv1142585544:85 , srv1157683362:85 , srv1194200830:85 , srv1281383465:85 , srv1296833299:85 , srv1456191102:85 , srv1594313192:85 , srv1821445845:85 , srv184683603:85 , srv1857728177:85 , srv1901856678:85 , srv1937075575:85 , srv1951787487:85 , srv1976207523:85 , srv1976311830:85 , srv1995773489:85 , srv2086374528:85 , srv2109889199:85 , srv211014239:85 , srv243066495:85 , srv415252475:85 , srv482791625:85 , srv498037065:85 , srv52890093:85 , srv557774186:85 , srv579224863:85 , srv580638772:85 , srv592455550:85 , srv606463801:85 , srv699217932:85 , srv72193591:85 , srv756991529:85 , srv78867977:85 , srv795147856:85 , srv823208837:85 , srv867705526:85 , srv410832460:652 } 2023-10-09 06:53:30,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:53:30,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1192272112=13, srv218358243=61, srv1281383465=18, srv159862055=33, srv161827955=34, srv78867977=90, srv146074749=25, srv1918889029=47, srv1995773489=53, srv1003842088=0, srv482791625=69, srv1433809142=23, srv1296833299=19, srv1010713630=1, srv823208837=92, srv1462261701=26, srv1901856678=46, srv52890093=72, srv592455550=78, srv2131072036=59, srv1870338922=43, srv239780194=62, srv1142585544=10, srv1056165619=4, srv1172059746=12, srv1857728177=42, srv580638772=77, srv2147017256=60, srv1510361780=29, srv1885545551=44, srv1976311830=51, srv405049219=66, srv6563482=80, srv776057433=88, srv66971886=82, srv563080538=75, srv1029165728=3, srv2086374528=56, srv867705526=93, srv378025476=65, srv515122492=71, srv498037065=70, srv706075030=84, srv1901503834=45, srv1734811335=39, srv788008886=89, srv1821445845=40, srv149080593=27, srv756991529=86, srv1422612740=22, srv1243092650=17, srv1976207523=50, srv952944997=97, srv775209016=87, srv579224863=76, srv920216613=95, srv1061392174=5, srv1570578810=30, srv1125560578=8, srv184683603=41, srv1684425784=37, srv658225910=81, srv283521566=64, srv72193591=85, srv211014239=58, srv1951787487=49, srv1366950860=20, srv1681862375=36, srv1157683362=11, srv1694005180=38, srv1194200830=14, srv1117176008=7, srv795147856=91, srv559943641=74, srv1937075575=48, srv1092366914=6, srv1197528851=15, srv981360614=99, srv1667673256=35, srv2076718082=54, srv243066495=63, srv606463801=79, srv699217932=83, srv1576879099=31, srv2084820580=55, srv967971771=98, srv1456191102=24, srv415252475=68, srv1594313192=32, srv557774186=73, srv1392776121=21, srv1977312673=52, srv410832460=67, srv1139136941=9, srv1207062693=16, srv2109889199=57, srv947733639=96, srv1017578764=2, srv1496601302=28, srv910327457=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:53:30,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-09 06:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-09 06:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-09 06:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-09 06:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-09 06:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-09 06:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-09 06:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-09 06:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-09 06:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-09 06:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-09 06:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-09 06:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-09 06:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-09 06:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-09 06:53:30,492 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7182857106794416E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:30,492 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv218358243:82 , srv1197528851:83 , srv1510361780:83 , srv161827955:83 , srv1694005180:83 , srv658225910:83 , srv967971771:83 , srv1003842088:84 , srv1010713630:84 , srv1017578764:84 , srv1029165728:84 , srv1061392174:84 , srv1139136941:84 , srv1172059746:84 , srv1192272112:84 , srv1207062693:84 , srv1243092650:84 , srv1366950860:84 , srv1392776121:84 , srv1422612740:84 , srv1433809142:84 , srv146074749:84 , srv1462261701:84 , srv149080593:84 , srv1496601302:84 , srv1570578810:84 , srv1576879099:84 , srv159862055:84 , srv1667673256:84 , srv1681862375:84 , srv1684425784:84 , srv1734811335:84 , srv1870338922:84 , srv1885545551:84 , srv1901503834:84 , srv1918889029:84 , srv1977312673:84 , srv2076718082:84 , srv2084820580:84 , srv2131072036:84 , srv2147017256:84 , srv239780194:84 , srv283521566:84 , srv378025476:84 , srv405049219:84 , srv515122492:84 , srv559943641:84 , srv563080538:84 , srv6563482:84 , srv66971886:84 , srv706075030:84 , srv775209016:84 , srv776057433:84 , srv788008886:84 , srv910327457:84 , srv920216613:84 , srv947733639:84 , srv952944997:84 , srv981360614:84 , srv1056165619:85 , srv1092366914:85 , srv1117176008:85 , srv1125560578:85 , srv1142585544:85 , srv1157683362:85 , srv1194200830:85 , srv1281383465:85 , srv1296833299:85 , srv1456191102:85 , srv1594313192:85 , srv1821445845:85 , srv184683603:85 , srv1857728177:85 , srv1901856678:85 , srv1937075575:85 , srv1951787487:85 , srv1976207523:85 , srv1976311830:85 , srv1995773489:85 , srv2086374528:85 , srv2109889199:85 , srv211014239:85 , srv243066495:85 , srv415252475:85 , srv482791625:85 , srv498037065:85 , srv52890093:85 , srv557774186:85 , srv579224863:85 , srv580638772:85 , srv592455550:85 , srv606463801:85 , srv699217932:85 , srv72193591:85 , srv756991529:85 , srv78867977:85 , srv795147856:85 , srv823208837:85 , srv867705526:85 , srv410832460:652 } 2023-10-09 06:53:30,529 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=291 (was 246) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=9180 (was 9173) - AvailableMemoryMB LEAK? - 2023-10-09 06:53:30,540 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=291, ProcessCount=170, AvailableMemoryMB=9180 2023-10-09 06:53:30,541 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-09 06:53:30,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-09 06:53:30,555 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1242138195:84 , srv1262310239:84 , srv1395543371:84 , srv1429631518:84 , srv1431193869:84 , srv1446789723:84 , srv1507295954:84 , srv1556032874:84 , srv1611958123:84 , srv1739657716:84 , srv1756949848:84 , srv1798501945:84 , srv1884461202:84 , srv1889190298:84 , srv1937271044:84 , srv2062833267:84 , srv2086890811:84 , srv2095421817:84 , srv274471346:84 , srv325220913:84 , srv364591054:84 , srv386563239:84 , srv454027715:84 , srv486301647:84 , srv500595912:84 , srv527713326:84 , srv720936560:84 , srv851678159:84 , srv945903299:84 , srv1360668183:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-09 06:53:30,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:53:30,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242138195=0, srv274471346=19, srv1884461202=13, srv1611958123=9, srv454027715=23, srv1429631518=4, srv1395543371=3, srv1262310239=1, srv1798501945=12, srv1431193869=5, srv1937271044=15, srv486301647=24, srv945903299=29, srv527713326=26, srv325220913=20, srv1739657716=10, srv1889190298=14, srv1360668183=2, srv1446789723=6, srv1756949848=11, srv2095421817=18, srv364591054=21, srv720936560=27, srv1507295954=7, srv386563239=22, srv2086890811=17, srv851678159=28, srv2062833267=16, srv1556032874=8, srv500595912=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-09 06:53:30,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1809904360808245, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:53:30,598 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912763338218488, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1809904360808245, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-09 06:55:30,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5930095 different iterations. Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912763338218488 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:55:30,584 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1242138195:90 , srv1262310239:90 , srv1360668183:90 , srv1395543371:90 , srv1429631518:90 , srv1431193869:90 , srv1446789723:90 , srv1507295954:90 , srv1556032874:90 , srv1611958123:90 , srv1739657716:90 , srv1756949848:90 , srv1798501945:90 , srv1884461202:90 , srv1889190298:90 , srv1937271044:90 , srv2062833267:90 , srv2086890811:90 , srv2095421817:90 , srv274471346:90 , srv325220913:90 , srv364591054:90 , srv386563239:90 , srv454027715:90 , srv486301647:90 , srv500595912:90 , srv527713326:90 , srv720936560:90 , srv851678159:90 , srv945903299:90 } 2023-10-09 06:55:30,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-09 06:55:30,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242138195=0, srv274471346=19, srv1884461202=13, srv1611958123=9, srv454027715=23, srv1429631518=4, srv1395543371=3, srv1262310239=1, srv1798501945=12, srv1431193869=5, srv1937271044=15, srv486301647=24, srv945903299=29, srv527713326=26, srv325220913=20, srv1739657716=10, srv1889190298=14, srv1360668183=2, srv1446789723=6, srv1756949848=11, srv2095421817=18, srv364591054=21, srv720936560=27, srv1507295954=7, srv386563239=22, srv2086890811=17, srv851678159=28, srv2062833267=16, srv1556032874=8, srv500595912=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-09 06:55:30,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-09 06:55:30,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-09 06:55:30,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-09 06:55:30,606 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-09 06:55:30,607 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1242138195:90 , srv1262310239:90 , srv1360668183:90 , srv1395543371:90 , srv1429631518:90 , srv1431193869:90 , srv1446789723:90 , srv1507295954:90 , srv1556032874:90 , srv1611958123:90 , srv1739657716:90 , srv1756949848:90 , srv1798501945:90 , srv1884461202:90 , srv1889190298:90 , srv1937271044:90 , srv2062833267:90 , srv2086890811:90 , srv2095421817:90 , srv274471346:90 , srv325220913:90 , srv364591054:90 , srv386563239:90 , srv454027715:90 , srv486301647:90 , srv500595912:90 , srv527713326:90 , srv720936560:90 , srv851678159:90 , srv945903299:90 } 2023-10-09 06:55:30,619 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=310 (was 291) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 170), AvailableMemoryMB=8241 (was 9180)