2023-10-19 16:53:26,239 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-19 16:53:26,547 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-19 16:53:26,645 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-19 16:53:26,712 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 16:53:26,722 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-19 16:53:26,736 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=293, ProcessCount=170, AvailableMemoryMB=11298 2023-10-19 16:53:26,738 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 16:53:26,738 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-19 16:53:26,866 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1024172716:84 , srv1051590673:84 , srv1061156963:84 , srv1091523002:84 , srv1125603721:84 , srv1142446094:84 , srv1144804624:84 , srv1151491911:84 , srv1200054829:84 , srv1208799926:84 , srv1209104679:84 , srv123637236:84 , srv1249635463:84 , srv1274751014:84 , srv1281959232:84 , srv1333907417:84 , srv1353612981:84 , srv1383149119:84 , srv138795163:84 , srv1409488413:84 , srv1497453913:84 , srv1527318517:84 , srv1556792107:84 , srv1590851666:84 , srv1616896906:84 , srv1630962973:84 , srv1668189741:84 , srv1699660150:84 , srv1713029464:84 , srv1718332007:84 , srv1743532334:84 , srv1759127300:84 , srv1771245867:84 , srv1784869225:84 , srv1787125588:84 , srv1816275865:84 , srv1824554062:84 , srv1838757535:84 , srv184946856:84 , srv1858419293:84 , srv1859273059:84 , srv1869499216:84 , srv1871977605:84 , srv1882612472:84 , srv188957158:84 , srv1891156700:84 , srv1898795424:84 , srv1913825701:84 , srv1914222847:84 , srv1961771141:84 , srv1967555937:84 , srv1970903003:84 , srv1978815075:84 , srv2008746869:84 , srv2009906162:84 , srv202164717:84 , srv2037529864:84 , srv2054124697:84 , srv2057728917:84 , srv2075059965:84 , srv2086248110:84 , srv2102482257:84 , srv25513248:84 , srv267747858:84 , srv284815003:84 , srv339334589:84 , srv343692754:84 , srv427965505:84 , srv433189012:84 , srv463341036:84 , srv505059365:84 , srv541139318:84 , srv568721990:84 , srv626209318:84 , srv634025389:84 , srv634060156:84 , srv635033489:84 , srv644981663:84 , srv667259891:84 , srv683002387:84 , srv694148197:84 , srv701534722:84 , srv710110185:84 , srv714255082:84 , srv716113397:84 , srv738953966:84 , srv764912922:84 , srv775763543:84 , srv824098860:84 , srv841980730:84 , srv847266707:84 , srv854817748:84 , srv858627891:84 , srv88158177:84 , srv897264267:84 , srv954630285:84 , srv959040555:84 , srv964216440:84 , srv974805292:84 , srv494752237:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-19 16:53:26,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:26,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2009906162=54, srv463341036=69, srv644981663=78, srv1759127300=31, srv710110185=83, srv1787125588=34, srv1967555937=50, srv1913825701=47, srv954630285=96, srv1970903003=51, srv775763543=88, srv267747858=63, srv1274751014=13, srv138795163=18, srv1616896906=24, srv694148197=81, srv1838757535=37, srv1882612472=43, srv1891156700=45, srv123637236=11, srv2008746869=53, srv1249635463=12, srv1590851666=23, srv1383149119=17, srv1859273059=40, srv738953966=86, srv505059365=71, srv2037529864=56, srv847266707=91, srv541139318=72, srv1771245867=32, srv1144804624=6, srv1353612981=16, srv184946856=38, srv188957158=44, srv284815003=64, srv494752237=70, srv959040555=97, srv1858419293=39, srv343692754=66, srv1961771141=49, srv1824554062=36, srv25513248=62, srv667259891=79, srv1051590673=1, srv716113397=85, srv683002387=80, srv854817748=92, srv1091523002=3, srv1151491911=7, srv433189012=68, srv1978815075=52, srv824098860=89, srv1556792107=22, srv1914222847=48, srv1209104679=10, srv2057728917=58, srv974805292=99, srv88158177=94, srv1743532334=30, srv339334589=65, srv858627891=93, srv1281959232=14, srv202164717=55, srv841980730=90, srv1125603721=4, srv2086248110=60, srv1024172716=0, srv1497453913=20, srv634060156=76, srv1816275865=35, srv634025389=75, srv1200054829=8, srv2102482257=61, srv897264267=95, srv764912922=87, srv1784869225=33, srv1898795424=46, srv2075059965=59, srv1527318517=21, srv1869499216=41, srv1061156963=2, srv1630962973=25, srv568721990=73, srv964216440=98, srv1713029464=28, srv1871977605=42, srv427965505=67, srv701534722=82, srv1142446094=5, srv1208799926=9, srv1333907417=15, srv1718332007=29, srv1699660150=27, srv635033489=77, srv1668189741=26, srv626209318=74, srv2054124697=57, srv1409488413=19, srv714255082=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:26,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:26,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:26,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:26,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:26,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:26,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:26,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:26,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:26,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 16:53:26,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 16:53:26,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 16:53:26,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 16:53:26,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 16:53:26,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 16:53:26,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 16:53:26,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 16:53:26,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 16:53:26,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 16:53:26,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 16:53:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 16:53:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 16:53:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 16:53:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 16:53:27,153 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:27,154 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 16:53:27,154 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 16:53:27,593 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 585 ms to try 5000 different iterations. Found a solution that moves 4734 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17356582232246015. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03990198741386206, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05828256569098627, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.526, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03990198741386206, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16891666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:27,617 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1091523002:83 , srv1699660150:83 , srv184946856:83 , srv1869499216:83 , srv2037529864:83 , srv2086248110:83 , srv897264267:83 , srv1274751014:84 , srv1497453913:84 , srv1838757535:84 , srv683002387:84 , srv1125603721:85 , srv1249635463:85 , srv1281959232:85 , srv1616896906:85 , srv1668189741:85 , srv1713029464:85 , srv1759127300:85 , srv1784869225:85 , srv1858419293:85 , srv1859273059:85 , srv188957158:85 , srv1898795424:85 , srv1914222847:85 , srv1961771141:85 , srv1978815075:85 , srv202164717:85 , srv2057728917:85 , srv634060156:85 , srv694148197:85 , srv847266707:85 , srv959040555:85 , srv974805292:85 , srv1024172716:86 , srv1144804624:86 , srv1200054829:86 , srv1527318517:86 , srv1556792107:86 , srv1718332007:86 , srv1771245867:86 , srv1816275865:86 , srv1891156700:86 , srv1913825701:86 , srv1967555937:86 , srv1970903003:86 , srv2075059965:86 , srv2102482257:86 , srv463341036:86 , srv505059365:86 , srv541139318:86 , srv626209318:86 , srv710110185:86 , srv716113397:86 , srv764912922:86 , srv824098860:86 , srv841980730:86 , srv858627891:86 , srv88158177:86 , srv954630285:86 , srv1061156963:87 , srv1142446094:87 , srv123637236:87 , srv1409488413:87 , srv1630962973:87 , srv1882612472:87 , srv2008746869:87 , srv2009906162:87 , srv2054124697:87 , srv267747858:87 , srv433189012:87 , srv634025389:87 , srv635033489:87 , srv644981663:87 , srv667259891:87 , srv854817748:87 , srv964216440:87 , srv1209104679:88 , srv1383149119:88 , srv138795163:88 , srv1743532334:88 , srv25513248:88 , srv284815003:88 , srv343692754:88 , srv738953966:88 , srv1353612981:89 , srv1824554062:89 , srv1871977605:89 , srv339334589:89 , srv701534722:89 , srv714255082:89 , srv1051590673:90 , srv1151491911:90 , srv427965505:90 , srv568721990:90 , srv775763543:90 , srv1208799926:91 , srv1333907417:91 , srv1590851666:91 , srv1787125588:92 , srv494752237:445 } 2023-10-19 16:53:27,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:27,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2009906162=54, srv463341036=69, srv644981663=78, srv1759127300=31, srv710110185=83, srv1787125588=34, srv1967555937=50, srv1913825701=47, srv954630285=96, srv1970903003=51, srv775763543=88, srv267747858=63, srv1274751014=13, srv138795163=18, srv1616896906=24, srv694148197=81, srv1838757535=37, srv1882612472=43, srv1891156700=45, srv123637236=11, srv2008746869=53, srv1249635463=12, srv1590851666=23, srv1383149119=17, srv1859273059=40, srv738953966=86, srv505059365=71, srv2037529864=56, srv847266707=91, srv541139318=72, srv1771245867=32, srv1144804624=6, srv1353612981=16, srv184946856=38, srv188957158=44, srv284815003=64, srv494752237=70, srv959040555=97, srv1858419293=39, srv343692754=66, srv1961771141=49, srv1824554062=36, srv25513248=62, srv667259891=79, srv1051590673=1, srv716113397=85, srv683002387=80, srv854817748=92, srv1091523002=3, srv1151491911=7, srv433189012=68, srv1978815075=52, srv824098860=89, srv1556792107=22, srv1914222847=48, srv1209104679=10, srv2057728917=58, srv974805292=99, srv88158177=94, srv1743532334=30, srv339334589=65, srv858627891=93, srv1281959232=14, srv202164717=55, srv841980730=90, srv1125603721=4, srv2086248110=60, srv1024172716=0, srv1497453913=20, srv634060156=76, srv1816275865=35, srv634025389=75, srv1200054829=8, srv2102482257=61, srv897264267=95, srv764912922=87, srv1784869225=33, srv1898795424=46, srv2075059965=59, srv1527318517=21, srv1869499216=41, srv1061156963=2, srv1630962973=25, srv568721990=73, srv964216440=98, srv1713029464=28, srv1871977605=42, srv427965505=67, srv701534722=82, srv1142446094=5, srv1208799926=9, srv1333907417=15, srv1718332007=29, srv1699660150=27, srv635033489=77, srv1668189741=26, srv626209318=74, srv2054124697=57, srv1409488413=19, srv714255082=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:27,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:27,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:27,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:27,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:27,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:27,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:27,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:27,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:27,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 16:53:27,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 16:53:27,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 16:53:27,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 16:53:27,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 16:53:27,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 16:53:27,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 16:53:27,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 16:53:27,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 16:53:27,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 16:53:27,782 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03990198741386206, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05828256569098627, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03990198741386206, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16891666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:27,782 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 16:53:27,783 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17353267414596996, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03990198741386206, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05828256569098627, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03990198741386206, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16891666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 16:53:28,093 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 424 ms to try 5000 different iterations. Found a solution that moves 2618 regions; Going from a computed imbalance of 0.17353267414596996 to a new imbalance of 0.015074656329492116. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022144082597030244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03612858939872464, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29088888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022144082597030244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.050083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:28,107 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1787125588:78 , srv1151491911:81 , srv1208799926:81 , srv1590851666:81 , srv1333907417:82 , srv701534722:82 , srv775763543:82 , srv1630962973:83 , srv343692754:83 , srv427965505:83 , srv714255082:83 , srv1051590673:84 , srv1142446094:84 , srv1209104679:84 , srv1353612981:84 , srv1383149119:84 , srv138795163:84 , srv1409488413:84 , srv1718332007:84 , srv1824554062:84 , srv2008746869:84 , srv284815003:84 , srv339334589:84 , srv433189012:84 , srv541139318:84 , srv568721990:84 , srv634025389:84 , srv710110185:84 , srv964216440:84 , srv1024172716:85 , srv1061156963:85 , srv1144804624:85 , srv1743532334:85 , srv1871977605:85 , srv1882612472:85 , srv188957158:85 , srv1913825701:85 , srv2009906162:85 , srv2054124697:85 , srv2075059965:85 , srv25513248:85 , srv267747858:85 , srv505059365:85 , srv635033489:85 , srv644981663:85 , srv667259891:85 , srv716113397:85 , srv764912922:85 , srv841980730:85 , srv1200054829:86 , srv123637236:86 , srv1249635463:86 , srv1497453913:86 , srv1527318517:86 , srv1556792107:86 , srv1759127300:86 , srv1771245867:86 , srv1816275865:86 , srv1859273059:86 , srv1898795424:86 , srv1967555937:86 , srv1970903003:86 , srv1978815075:86 , srv463341036:86 , srv738953966:86 , srv854817748:86 , srv88158177:86 , srv1125603721:87 , srv1274751014:87 , srv1616896906:87 , srv1668189741:87 , srv1858419293:87 , srv1891156700:87 , srv2086248110:87 , srv2102482257:87 , srv626209318:87 , srv824098860:87 , srv858627891:87 , srv959040555:87 , srv974805292:87 , srv1281959232:88 , srv1713029464:88 , srv1869499216:88 , srv1914222847:88 , srv1961771141:88 , srv202164717:88 , srv2037529864:88 , srv2057728917:88 , srv634060156:88 , srv683002387:88 , srv847266707:88 , srv1091523002:89 , srv1784869225:89 , srv184946856:89 , srv694148197:89 , srv954630285:89 , srv1699660150:90 , srv1838757535:90 , srv897264267:91 , srv494752237:526 } 2023-10-19 16:53:28,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:28,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2009906162=54, srv463341036=69, srv644981663=78, srv1759127300=31, srv710110185=83, srv1787125588=34, srv1967555937=50, srv1913825701=47, srv954630285=96, srv1970903003=51, srv775763543=88, srv267747858=63, srv1274751014=13, srv138795163=18, srv1616896906=24, srv694148197=81, srv1838757535=37, srv1882612472=43, srv1891156700=45, srv123637236=11, srv2008746869=53, srv1249635463=12, srv1590851666=23, srv1383149119=17, srv1859273059=40, srv738953966=86, srv505059365=71, srv2037529864=56, srv847266707=91, srv541139318=72, srv1771245867=32, srv1144804624=6, srv1353612981=16, srv184946856=38, srv188957158=44, srv284815003=64, srv494752237=70, srv959040555=97, srv1858419293=39, srv343692754=66, srv1961771141=49, srv1824554062=36, srv25513248=62, srv667259891=79, srv1051590673=1, srv716113397=85, srv683002387=80, srv854817748=92, srv1091523002=3, srv1151491911=7, srv433189012=68, srv1978815075=52, srv824098860=89, srv1556792107=22, srv1914222847=48, srv1209104679=10, srv2057728917=58, srv974805292=99, srv88158177=94, srv1743532334=30, srv339334589=65, srv858627891=93, srv1281959232=14, srv202164717=55, srv841980730=90, srv1125603721=4, srv2086248110=60, srv1024172716=0, srv1497453913=20, srv634060156=76, srv1816275865=35, srv634025389=75, srv1200054829=8, srv2102482257=61, srv897264267=95, srv764912922=87, srv1784869225=33, srv1898795424=46, srv2075059965=59, srv1527318517=21, srv1869499216=41, srv1061156963=2, srv1630962973=25, srv568721990=73, srv964216440=98, srv1713029464=28, srv1871977605=42, srv427965505=67, srv701534722=82, srv1142446094=5, srv1208799926=9, srv1333907417=15, srv1718332007=29, srv1699660150=27, srv635033489=77, srv1668189741=26, srv626209318=74, srv2054124697=57, srv1409488413=19, srv714255082=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:28,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:28,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:28,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 16:53:28,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 16:53:28,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 16:53:28,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 16:53:28,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 16:53:28,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 16:53:28,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 16:53:28,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 16:53:28,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 16:53:28,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 16:53:28,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 16:53:28,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 16:53:28,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 16:53:28,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 16:53:28,289 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022144082597030244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03612858939872464, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022144082597030244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.050083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:28,289 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 16:53:28,289 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015056324701682378, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022144082597030244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03612858939872464, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022144082597030244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.050083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 16:53:28,558 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 383 ms to try 5000 different iterations. Found a solution that moves 998 regions; Going from a computed imbalance of 0.015056324701682378 to a new imbalance of 7.647384398216562E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006547761159855293); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011331154474650633); MoveCostFunction : (multiplier=7.0, imbalance=0.11088888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006547761159855293); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:28,562 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv954630285:82 , srv1784869225:83 , srv1838757535:83 , srv568721990:83 , srv738953966:83 , srv1051590673:84 , srv123637236:84 , srv1281959232:84 , srv1743532334:84 , srv1871977605:84 , srv1891156700:84 , srv1914222847:84 , srv202164717:84 , srv2057728917:84 , srv634060156:84 , srv694148197:84 , srv775763543:84 , srv847266707:84 , srv854817748:84 , srv897264267:84 , srv1091523002:85 , srv1200054829:85 , srv1208799926:85 , srv1333907417:85 , srv1527318517:85 , srv1616896906:85 , srv1668189741:85 , srv1713029464:85 , srv1771245867:85 , srv1816275865:85 , srv1824554062:85 , srv1858419293:85 , srv1859273059:85 , srv1882612472:85 , srv1961771141:85 , srv2054124697:85 , srv2102482257:85 , srv25513248:85 , srv267747858:85 , srv284815003:85 , srv339334589:85 , srv427965505:85 , srv433189012:85 , srv626209318:85 , srv635033489:85 , srv644981663:85 , srv683002387:85 , srv824098860:85 , srv858627891:85 , srv88158177:85 , srv959040555:85 , srv1024172716:86 , srv1061156963:86 , srv1125603721:86 , srv1144804624:86 , srv1209104679:86 , srv1249635463:86 , srv1274751014:86 , srv1353612981:86 , srv1383149119:86 , srv138795163:86 , srv1409488413:86 , srv1556792107:86 , srv1630962973:86 , srv1699660150:86 , srv1759127300:86 , srv184946856:86 , srv1967555937:86 , srv1970903003:86 , srv1978815075:86 , srv2008746869:86 , srv2009906162:86 , srv2037529864:86 , srv343692754:86 , srv463341036:86 , srv505059365:86 , srv667259891:86 , srv701534722:86 , srv714255082:86 , srv764912922:86 , srv841980730:86 , srv964216440:86 , srv1142446094:87 , srv1151491911:87 , srv1497453913:87 , srv1590851666:87 , srv1718332007:87 , srv1787125588:87 , srv1869499216:87 , srv1898795424:87 , srv1913825701:87 , srv2075059965:87 , srv2086248110:87 , srv541139318:87 , srv634025389:87 , srv710110185:87 , srv716113397:87 , srv974805292:87 , srv188957158:88 , srv494752237:545 } 2023-10-19 16:53:28,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:28,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2009906162=54, srv463341036=69, srv644981663=78, srv1759127300=31, srv710110185=83, srv1787125588=34, srv1967555937=50, srv1913825701=47, srv954630285=96, srv1970903003=51, srv775763543=88, srv267747858=63, srv1274751014=13, srv138795163=18, srv1616896906=24, srv694148197=81, srv1838757535=37, srv1882612472=43, srv1891156700=45, srv123637236=11, srv2008746869=53, srv1249635463=12, srv1590851666=23, srv1383149119=17, srv1859273059=40, srv738953966=86, srv505059365=71, srv2037529864=56, srv847266707=91, srv541139318=72, srv1771245867=32, srv1144804624=6, srv1353612981=16, srv184946856=38, srv188957158=44, srv284815003=64, srv494752237=70, srv959040555=97, srv1858419293=39, srv343692754=66, srv1961771141=49, srv1824554062=36, srv25513248=62, srv667259891=79, srv1051590673=1, srv716113397=85, srv683002387=80, srv854817748=92, srv1091523002=3, srv1151491911=7, srv433189012=68, srv1978815075=52, srv824098860=89, srv1556792107=22, srv1914222847=48, srv1209104679=10, srv2057728917=58, srv974805292=99, srv88158177=94, srv1743532334=30, srv339334589=65, srv858627891=93, srv1281959232=14, srv202164717=55, srv841980730=90, srv1125603721=4, srv2086248110=60, srv1024172716=0, srv1497453913=20, srv634060156=76, srv1816275865=35, srv634025389=75, srv1200054829=8, srv2102482257=61, srv897264267=95, srv764912922=87, srv1784869225=33, srv1898795424=46, srv2075059965=59, srv1527318517=21, srv1869499216=41, srv1061156963=2, srv1630962973=25, srv568721990=73, srv964216440=98, srv1713029464=28, srv1871977605=42, srv427965505=67, srv701534722=82, srv1142446094=5, srv1208799926=9, srv1333907417=15, srv1718332007=29, srv1699660150=27, srv635033489=77, srv1668189741=26, srv626209318=74, srv2054124697=57, srv1409488413=19, srv714255082=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:28,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:28,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:28,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 16:53:28,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 16:53:28,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 16:53:28,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 16:53:28,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 16:53:28,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 16:53:28,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 16:53:28,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 16:53:28,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 16:53:28,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 16:53:28,706 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006547761159855293); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011331154474650633); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006547761159855293); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:28,706 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 16:53:28,706 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.577502944610306E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006547761159855293); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011331154474650633); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006547761159855293); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 16:53:29,044 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations. Found a solution that moves 254 regions; Going from a computed imbalance of 7.577502944610306E-4 to a new imbalance of 2.7177002912844625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.02822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:29,045 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1353612981:83 , srv1590851666:83 , srv1699660150:83 , srv188957158:83 , srv974805292:83 , srv1051590673:84 , srv1061156963:84 , srv1125603721:84 , srv1142446094:84 , srv1151491911:84 , srv1209104679:84 , srv1249635463:84 , srv1333907417:84 , srv1383149119:84 , srv138795163:84 , srv1556792107:84 , srv1713029464:84 , srv1718332007:84 , srv1824554062:84 , srv184946856:84 , srv1869499216:84 , srv1871977605:84 , srv1898795424:84 , srv1913825701:84 , srv1961771141:84 , srv1967555937:84 , srv1970903003:84 , srv2009906162:84 , srv2075059965:84 , srv2102482257:84 , srv25513248:84 , srv339334589:84 , srv427965505:84 , srv463341036:84 , srv626209318:84 , srv634025389:84 , srv634060156:84 , srv694148197:84 , srv701534722:84 , srv714255082:84 , srv716113397:84 , srv738953966:84 , srv824098860:84 , srv858627891:84 , srv897264267:84 , srv1024172716:85 , srv1091523002:85 , srv1144804624:85 , srv1200054829:85 , srv1274751014:85 , srv1281959232:85 , srv1409488413:85 , srv1497453913:85 , srv1527318517:85 , srv1616896906:85 , srv1743532334:85 , srv1759127300:85 , srv1771245867:85 , srv1784869225:85 , srv1787125588:85 , srv1816275865:85 , srv1838757535:85 , srv1858419293:85 , srv1882612472:85 , srv1891156700:85 , srv1914222847:85 , srv1978815075:85 , srv2008746869:85 , srv202164717:85 , srv2037529864:85 , srv2054124697:85 , srv2057728917:85 , srv2086248110:85 , srv267747858:85 , srv284815003:85 , srv343692754:85 , srv541139318:85 , srv568721990:85 , srv635033489:85 , srv644981663:85 , srv667259891:85 , srv683002387:85 , srv710110185:85 , srv764912922:85 , srv841980730:85 , srv847266707:85 , srv854817748:85 , srv88158177:85 , srv954630285:85 , srv959040555:85 , srv964216440:85 , srv1208799926:86 , srv123637236:86 , srv1630962973:86 , srv1668189741:86 , srv1859273059:86 , srv433189012:86 , srv505059365:86 , srv775763543:86 , srv494752237:627 } 2023-10-19 16:53:29,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:29,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2009906162=54, srv463341036=69, srv644981663=78, srv1759127300=31, srv710110185=83, srv1787125588=34, srv1967555937=50, srv1913825701=47, srv954630285=96, srv1970903003=51, srv775763543=88, srv267747858=63, srv1274751014=13, srv138795163=18, srv1616896906=24, srv694148197=81, srv1838757535=37, srv1882612472=43, srv1891156700=45, srv123637236=11, srv2008746869=53, srv1249635463=12, srv1590851666=23, srv1383149119=17, srv1859273059=40, srv738953966=86, srv505059365=71, srv2037529864=56, srv847266707=91, srv541139318=72, srv1771245867=32, srv1144804624=6, srv1353612981=16, srv184946856=38, srv188957158=44, srv284815003=64, srv494752237=70, srv959040555=97, srv1858419293=39, srv343692754=66, srv1961771141=49, srv1824554062=36, srv25513248=62, srv667259891=79, srv1051590673=1, srv716113397=85, srv683002387=80, srv854817748=92, srv1091523002=3, srv1151491911=7, srv433189012=68, srv1978815075=52, srv824098860=89, srv1556792107=22, srv1914222847=48, srv1209104679=10, srv2057728917=58, srv974805292=99, srv88158177=94, srv1743532334=30, srv339334589=65, srv858627891=93, srv1281959232=14, srv202164717=55, srv841980730=90, srv1125603721=4, srv2086248110=60, srv1024172716=0, srv1497453913=20, srv634060156=76, srv1816275865=35, srv634025389=75, srv1200054829=8, srv2102482257=61, srv897264267=95, srv764912922=87, srv1784869225=33, srv1898795424=46, srv2075059965=59, srv1527318517=21, srv1869499216=41, srv1061156963=2, srv1630962973=25, srv568721990=73, srv964216440=98, srv1713029464=28, srv1871977605=42, srv427965505=67, srv701534722=82, srv1142446094=5, srv1208799926=9, srv1333907417=15, srv1718332007=29, srv1699660150=27, srv635033489=77, srv1668189741=26, srv626209318=74, srv2054124697=57, srv1409488413=19, srv714255082=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:29,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:29,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:29,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 16:53:29,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 16:53:29,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 16:53:29,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 16:53:29,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 16:53:29,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 16:53:29,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 16:53:29,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 16:53:29,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 16:53:29,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 16:53:29,169 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:29,169 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 16:53:29,169 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.539845689921844E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 16:53:29,403 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 329 ms to try 5000 different iterations. Found a solution that moves 18 regions; Going from a computed imbalance of 2.539845689921844E-5 to a new imbalance of 7.815460494229802E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.002); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:29,403 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1208799926:83 , srv123637236:83 , srv1668189741:83 , srv505059365:83 , srv667259891:83 , srv1024172716:84 , srv1051590673:84 , srv1061156963:84 , srv1091523002:84 , srv1125603721:84 , srv1142446094:84 , srv1144804624:84 , srv1151491911:84 , srv1200054829:84 , srv1209104679:84 , srv1274751014:84 , srv1281959232:84 , srv1333907417:84 , srv1353612981:84 , srv1383149119:84 , srv138795163:84 , srv1409488413:84 , srv1497453913:84 , srv1527318517:84 , srv1556792107:84 , srv1590851666:84 , srv1616896906:84 , srv1630962973:84 , srv1699660150:84 , srv1713029464:84 , srv1743532334:84 , srv1759127300:84 , srv1771245867:84 , srv1784869225:84 , srv1787125588:84 , srv1816275865:84 , srv1824554062:84 , srv1838757535:84 , srv184946856:84 , srv1858419293:84 , srv1859273059:84 , srv1869499216:84 , srv1871977605:84 , srv1882612472:84 , srv1891156700:84 , srv1898795424:84 , srv1913825701:84 , srv1914222847:84 , srv1961771141:84 , srv1967555937:84 , srv1970903003:84 , srv1978815075:84 , srv2008746869:84 , srv2009906162:84 , srv202164717:84 , srv2037529864:84 , srv2054124697:84 , srv2057728917:84 , srv2075059965:84 , srv2086248110:84 , srv2102482257:84 , srv25513248:84 , srv267747858:84 , srv284815003:84 , srv339334589:84 , srv343692754:84 , srv427965505:84 , srv433189012:84 , srv463341036:84 , srv541139318:84 , srv568721990:84 , srv626209318:84 , srv634025389:84 , srv635033489:84 , srv644981663:84 , srv683002387:84 , srv694148197:84 , srv710110185:84 , srv714255082:84 , srv716113397:84 , srv764912922:84 , srv775763543:84 , srv824098860:84 , srv841980730:84 , srv847266707:84 , srv854817748:84 , srv858627891:84 , srv88158177:84 , srv897264267:84 , srv954630285:84 , srv959040555:84 , srv964216440:84 , srv974805292:84 , srv1249635463:85 , srv1718332007:85 , srv188957158:85 , srv634060156:85 , srv701534722:85 , srv738953966:85 , srv494752237:683 } 2023-10-19 16:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2009906162=54, srv463341036=69, srv644981663=78, srv1759127300=31, srv710110185=83, srv1787125588=34, srv1967555937=50, srv1913825701=47, srv954630285=96, srv1970903003=51, srv775763543=88, srv267747858=63, srv1274751014=13, srv138795163=18, srv1616896906=24, srv694148197=81, srv1838757535=37, srv1882612472=43, srv1891156700=45, srv123637236=11, srv2008746869=53, srv1249635463=12, srv1590851666=23, srv1383149119=17, srv1859273059=40, srv738953966=86, srv505059365=71, srv2037529864=56, srv847266707=91, srv541139318=72, srv1771245867=32, srv1144804624=6, srv1353612981=16, srv184946856=38, srv188957158=44, srv284815003=64, srv494752237=70, srv959040555=97, srv1858419293=39, srv343692754=66, srv1961771141=49, srv1824554062=36, srv25513248=62, srv667259891=79, srv1051590673=1, srv716113397=85, srv683002387=80, srv854817748=92, srv1091523002=3, srv1151491911=7, srv433189012=68, srv1978815075=52, srv824098860=89, srv1556792107=22, srv1914222847=48, srv1209104679=10, srv2057728917=58, srv974805292=99, srv88158177=94, srv1743532334=30, srv339334589=65, srv858627891=93, srv1281959232=14, srv202164717=55, srv841980730=90, srv1125603721=4, srv2086248110=60, srv1024172716=0, srv1497453913=20, srv634060156=76, srv1816275865=35, srv634025389=75, srv1200054829=8, srv2102482257=61, srv897264267=95, srv764912922=87, srv1784869225=33, srv1898795424=46, srv2075059965=59, srv1527318517=21, srv1869499216=41, srv1061156963=2, srv1630962973=25, srv568721990=73, srv964216440=98, srv1713029464=28, srv1871977605=42, srv427965505=67, srv701534722=82, srv1142446094=5, srv1208799926=9, srv1333907417=15, srv1718332007=29, srv1699660150=27, srv635033489=77, srv1668189741=26, srv626209318=74, srv2054124697=57, srv1409488413=19, srv714255082=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:29,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:29,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:29,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 16:53:29,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 16:53:29,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 16:53:29,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 16:53:29,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 16:53:29,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 16:53:29,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 16:53:29,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:29,538 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1208799926:83 , srv123637236:83 , srv1668189741:83 , srv505059365:83 , srv667259891:83 , srv1024172716:84 , srv1051590673:84 , srv1061156963:84 , srv1091523002:84 , srv1125603721:84 , srv1142446094:84 , srv1144804624:84 , srv1151491911:84 , srv1200054829:84 , srv1209104679:84 , srv1274751014:84 , srv1281959232:84 , srv1333907417:84 , srv1353612981:84 , srv1383149119:84 , srv138795163:84 , srv1409488413:84 , srv1497453913:84 , srv1527318517:84 , srv1556792107:84 , srv1590851666:84 , srv1616896906:84 , srv1630962973:84 , srv1699660150:84 , srv1713029464:84 , srv1743532334:84 , srv1759127300:84 , srv1771245867:84 , srv1784869225:84 , srv1787125588:84 , srv1816275865:84 , srv1824554062:84 , srv1838757535:84 , srv184946856:84 , srv1858419293:84 , srv1859273059:84 , srv1869499216:84 , srv1871977605:84 , srv1882612472:84 , srv1891156700:84 , srv1898795424:84 , srv1913825701:84 , srv1914222847:84 , srv1961771141:84 , srv1967555937:84 , srv1970903003:84 , srv1978815075:84 , srv2008746869:84 , srv2009906162:84 , srv202164717:84 , srv2037529864:84 , srv2054124697:84 , srv2057728917:84 , srv2075059965:84 , srv2086248110:84 , srv2102482257:84 , srv25513248:84 , srv267747858:84 , srv284815003:84 , srv339334589:84 , srv343692754:84 , srv427965505:84 , srv433189012:84 , srv463341036:84 , srv541139318:84 , srv568721990:84 , srv626209318:84 , srv634025389:84 , srv635033489:84 , srv644981663:84 , srv683002387:84 , srv694148197:84 , srv710110185:84 , srv714255082:84 , srv716113397:84 , srv764912922:84 , srv775763543:84 , srv824098860:84 , srv841980730:84 , srv847266707:84 , srv854817748:84 , srv858627891:84 , srv88158177:84 , srv897264267:84 , srv954630285:84 , srv959040555:84 , srv964216440:84 , srv974805292:84 , srv1249635463:85 , srv1718332007:85 , srv188957158:85 , srv634060156:85 , srv701534722:85 , srv738953966:85 , srv494752237:683 } 2023-10-19 16:53:29,567 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=293 (was 293), ProcessCount=170 (was 170), AvailableMemoryMB=10791 (was 11298) 2023-10-19 16:53:29,576 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=293, ProcessCount=170, AvailableMemoryMB=10790 2023-10-19 16:53:29,576 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 16:53:29,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-19 16:53:29,586 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1083525259:84 , srv1207788018:84 , srv1256391870:84 , srv1312853558:84 , srv1320192379:84 , srv1330653637:84 , srv1332153133:84 , srv1337482792:84 , srv1427956201:84 , srv1479890681:84 , srv153119107:84 , srv1613607323:84 , srv169684228:84 , srv1755429807:84 , srv1756102570:84 , srv1802978561:84 , srv1853175827:84 , srv1867260040:84 , srv2051270207:84 , srv2107145107:84 , srv283001827:84 , srv287163805:84 , srv359206463:84 , srv500311069:84 , srv59460618:84 , srv692171742:84 , srv7192381:84 , srv808347811:84 , srv818127578:84 , srv1769724185:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-19 16:53:29,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:53:29,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1083525259=0, srv500311069=24, srv808347811=28, srv692171742=26, srv1802978561=16, srv153119107=10, srv1312853558=3, srv169684228=12, srv818127578=29, srv1337482792=7, srv7192381=27, srv1756102570=14, srv2107145107=20, srv1207788018=1, srv1332153133=6, srv1867260040=18, srv1256391870=2, srv283001827=21, srv59460618=25, srv1755429807=13, srv1853175827=17, srv1427956201=8, srv287163805=22, srv359206463=23, srv1613607323=11, srv1769724185=15, srv2051270207=19, srv1320192379=4, srv1479890681=9, srv1330653637=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:53:29,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:53:29,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:53:29,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:53:29,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:53:29,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:53:29,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:53:29,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:53:29,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:53:29,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-19 16:53:29,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0928875311972701, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:53:29,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912485728842027, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0928875311972701, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-19 16:55:29,602 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6015563 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912485728842027 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:55:29,609 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1083525259:90 , srv1207788018:90 , srv1256391870:90 , srv1312853558:90 , srv1320192379:90 , srv1330653637:90 , srv1332153133:90 , srv1337482792:90 , srv1427956201:90 , srv1479890681:90 , srv153119107:90 , srv1613607323:90 , srv169684228:90 , srv1755429807:90 , srv1756102570:90 , srv1769724185:90 , srv1802978561:90 , srv1853175827:90 , srv1867260040:90 , srv2051270207:90 , srv2107145107:90 , srv283001827:90 , srv287163805:90 , srv359206463:90 , srv500311069:90 , srv59460618:90 , srv692171742:90 , srv7192381:90 , srv808347811:90 , srv818127578:90 } 2023-10-19 16:55:29,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 16:55:29,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1083525259=0, srv500311069=24, srv808347811=28, srv692171742=26, srv1802978561=16, srv153119107=10, srv1312853558=3, srv169684228=12, srv818127578=29, srv1337482792=7, srv7192381=27, srv1756102570=14, srv2107145107=20, srv1207788018=1, srv1332153133=6, srv1867260040=18, srv1256391870=2, srv283001827=21, srv59460618=25, srv1755429807=13, srv1853175827=17, srv1427956201=8, srv287163805=22, srv359206463=23, srv1613607323=11, srv1769724185=15, srv2051270207=19, srv1320192379=4, srv1479890681=9, srv1330653637=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 16:55:29,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 16:55:29,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 16:55:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 16:55:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 16:55:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 16:55:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 16:55:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 16:55:29,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-19 16:55:29,631 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 16:55:29,631 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1083525259:90 , srv1207788018:90 , srv1256391870:90 , srv1312853558:90 , srv1320192379:90 , srv1330653637:90 , srv1332153133:90 , srv1337482792:90 , srv1427956201:90 , srv1479890681:90 , srv153119107:90 , srv1613607323:90 , srv169684228:90 , srv1755429807:90 , srv1756102570:90 , srv1769724185:90 , srv1802978561:90 , srv1853175827:90 , srv1867260040:90 , srv2051270207:90 , srv2107145107:90 , srv283001827:90 , srv287163805:90 , srv359206463:90 , srv500311069:90 , srv59460618:90 , srv692171742:90 , srv7192381:90 , srv808347811:90 , srv818127578:90 } 2023-10-19 16:55:29,644 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=237 (was 293), ProcessCount=167 (was 170), AvailableMemoryMB=10568 (was 10790)