2023-02-13 11:53:44,748 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-13 11:53:45,024 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 11:53:45,116 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 11:53:45,166 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 11:53:45,173 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 11:53:45,185 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=169, AvailableMemoryMB=5918 2023-02-13 11:53:45,187 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 11:53:45,187 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 11:53:45,301 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1029669949:84 , srv1033124062:84 , srv1042680835:84 , srv1127882018:84 , srv1196329793:84 , srv1228779221:84 , srv1235591368:84 , srv1265940794:84 , srv1299753045:84 , srv1382749977:84 , srv1405137645:84 , srv1433733687:84 , srv1471965241:84 , srv1481052097:84 , srv1497057441:84 , srv1506994316:84 , srv1520502285:84 , srv1535760983:84 , srv1546650291:84 , srv1552513888:84 , srv1581717320:84 , srv1605798803:84 , srv1651539070:84 , srv1711073204:84 , srv1743734641:84 , srv1751982378:84 , srv1768119857:84 , srv1792265557:84 , srv179645158:84 , srv1807614208:84 , srv1828075982:84 , srv1849868235:84 , srv1875014949:84 , srv1921727002:84 , srv192390416:84 , srv1928909122:84 , srv1929812992:84 , srv1936553676:84 , srv1950911721:84 , srv1956425742:84 , srv199582432:84 , srv2022671429:84 , srv202958649:84 , srv2085445741:84 , srv2120472948:84 , srv2141602189:84 , srv2143047788:84 , srv223839949:84 , srv229899075:84 , srv239445402:84 , srv254659623:84 , srv289521678:84 , srv290812133:84 , srv294766880:84 , srv307640114:84 , srv313950399:84 , srv317512409:84 , srv322922283:84 , srv325225840:84 , srv331772870:84 , srv335385317:84 , srv357384470:84 , srv369936074:84 , srv393556716:84 , srv405324813:84 , srv469257556:84 , srv484251363:84 , srv494822386:84 , srv514099285:84 , srv599378060:84 , srv606831325:84 , srv612204437:84 , srv614511499:84 , srv638004106:84 , srv655865275:84 , srv667935927:84 , srv67534236:84 , srv68336041:84 , srv684994828:84 , srv68795667:84 , srv690294933:84 , srv700206220:84 , srv704017879:84 , srv704213595:84 , srv758402839:84 , srv761665995:84 , srv77011820:84 , srv773012907:84 , srv840828919:84 , srv841972787:84 , srv868053119:84 , srv868771167:84 , srv872338569:84 , srv928595276:84 , srv93120499:84 , srv941945316:84 , srv961347036:84 , srv98306469:84 , srv992704139:84 , srv673766745:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-13 11:53:45,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:45,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1506994316=15, srv928595276=94, srv357384470=61, srv992704139=99, srv1928909122=35, srv1581717320=20, srv469257556=65, srv322922283=57, srv1875014949=32, srv655865275=74, srv872338569=93, srv1546650291=18, srv614511499=72, srv93120499=95, srv1936553676=37, srv606831325=70, srv1382749977=9, srv179645158=28, srv1651539070=22, srv1956425742=39, srv773012907=88, srv961347036=97, srv1950911721=38, srv667935927=75, srv1228779221=5, srv1471965241=12, srv1921727002=33, srv1929812992=36, srv1751982378=25, srv229899075=48, srv294766880=53, srv494822386=67, srv673766745=76, srv1497057441=14, srv840828919=89, srv325225840=58, srv1196329793=4, srv1605798803=21, srv1481052097=13, srv313950399=55, srv2022671429=41, srv868771167=92, srv1535760983=17, srv393556716=63, srv307640114=54, srv599378060=69, srv1807614208=29, srv684994828=79, srv1127882018=3, srv1743734641=24, srv77011820=87, srv941945316=96, srv1029669949=0, srv331772870=59, srv1828075982=30, srv484251363=66, srv758402839=85, srv2120472948=44, srv223839949=47, srv1552513888=19, srv289521678=51, srv68336041=78, srv1520502285=16, srv1792265557=27, srv690294933=81, srv841972787=90, srv1768119857=26, srv612204437=71, srv704213595=84, srv1235591368=6, srv254659623=50, srv67534236=77, srv317512409=56, srv1042680835=2, srv638004106=73, srv704017879=83, srv290812133=52, srv514099285=68, srv239445402=49, srv405324813=64, srv868053119=91, srv68795667=80, srv2143047788=46, srv1405137645=10, srv369936074=62, srv761665995=86, srv1299753045=8, srv199582432=40, srv1433733687=11, srv202958649=42, srv2141602189=45, srv1711073204=23, srv192390416=34, srv98306469=98, srv1849868235=31, srv1033124062=1, srv1265940794=7, srv335385317=60, srv2085445741=43, srv700206220=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:45,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 11:53:45,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 11:53:45,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 11:53:45,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 11:53:45,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 11:53:45,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 11:53:45,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 11:53:45,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 11:53:45,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 11:53:45,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 11:53:45,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 11:53:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 11:53:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 11:53:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 11:53:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 11:53:45,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 11:53:45,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 11:53:45,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 11:53:45,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 11:53:45,651 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:45,652 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 11:53:45,652 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 11:53:45,970 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 532 ms to try 5000 different iterations. Found a solution that moves 4721 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17185094934558945. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03795955474558878, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05774863289785541, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5245555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03795955474558878, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23608333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:45,987 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1265940794:81 , srv1546650291:81 , srv1950911721:81 , srv331772870:81 , srv928595276:81 , srv1433733687:82 , srv1506994316:82 , srv1471965241:83 , srv1481052097:83 , srv1497057441:83 , srv1849868235:83 , srv1956425742:83 , srv335385317:83 , srv690294933:83 , srv77011820:83 , srv840828919:83 , srv1033124062:84 , srv1751982378:84 , srv1828075982:84 , srv294766880:84 , srv313950399:84 , srv758402839:84 , srv961347036:84 , srv1196329793:85 , srv1299753045:85 , srv1535760983:85 , srv1651539070:85 , srv192390416:85 , srv317512409:85 , srv514099285:85 , srv684994828:85 , srv704017879:85 , srv704213595:85 , srv841972787:85 , srv941945316:85 , srv1382749977:86 , srv1405137645:86 , srv1552513888:86 , srv1792265557:86 , srv199582432:86 , srv223839949:86 , srv254659623:86 , srv469257556:86 , srv614511499:86 , srv67534236:86 , srv68795667:86 , srv773012907:86 , srv992704139:86 , srv1042680835:87 , srv1743734641:87 , srv179645158:87 , srv1807614208:87 , srv1936553676:87 , srv2085445741:87 , srv239445402:87 , srv290812133:87 , srv322922283:87 , srv484251363:87 , srv606831325:87 , srv612204437:87 , srv655865275:87 , srv761665995:87 , srv872338569:87 , srv1711073204:88 , srv1768119857:88 , srv1921727002:88 , srv1928909122:88 , srv2141602189:88 , srv325225840:88 , srv357384470:88 , srv667935927:88 , srv68336041:88 , srv700206220:88 , srv868771167:88 , srv93120499:88 , srv1235591368:89 , srv1875014949:89 , srv1929812992:89 , srv2022671429:89 , srv369936074:89 , srv599378060:89 , srv868053119:89 , srv98306469:89 , srv1228779221:90 , srv202958649:90 , srv2143047788:90 , srv229899075:90 , srv289521678:90 , srv393556716:90 , srv638004106:90 , srv1520502285:91 , srv1581717320:91 , srv1605798803:91 , srv307640114:91 , srv494822386:91 , srv2120472948:92 , srv405324813:92 , srv1127882018:94 , srv1029669949:95 , srv673766745:427 } 2023-02-13 11:53:45,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:45,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1506994316=15, srv928595276=94, srv357384470=61, srv992704139=99, srv1928909122=35, srv1581717320=20, srv469257556=65, srv322922283=57, srv1875014949=32, srv655865275=74, srv872338569=93, srv1546650291=18, srv614511499=72, srv93120499=95, srv1936553676=37, srv606831325=70, srv1382749977=9, srv179645158=28, srv1651539070=22, srv1956425742=39, srv773012907=88, srv961347036=97, srv1950911721=38, srv667935927=75, srv1228779221=5, srv1471965241=12, srv1921727002=33, srv1929812992=36, srv1751982378=25, srv229899075=48, srv294766880=53, srv494822386=67, srv673766745=76, srv1497057441=14, srv840828919=89, srv325225840=58, srv1196329793=4, srv1605798803=21, srv1481052097=13, srv313950399=55, srv2022671429=41, srv868771167=92, srv1535760983=17, srv393556716=63, srv307640114=54, srv599378060=69, srv1807614208=29, srv684994828=79, srv1127882018=3, srv1743734641=24, srv77011820=87, srv941945316=96, srv1029669949=0, srv331772870=59, srv1828075982=30, srv484251363=66, srv758402839=85, srv2120472948=44, srv223839949=47, srv1552513888=19, srv289521678=51, srv68336041=78, srv1520502285=16, srv1792265557=27, srv690294933=81, srv841972787=90, srv1768119857=26, srv612204437=71, srv704213595=84, srv1235591368=6, srv254659623=50, srv67534236=77, srv317512409=56, srv1042680835=2, srv638004106=73, srv704017879=83, srv290812133=52, srv514099285=68, srv239445402=49, srv405324813=64, srv868053119=91, srv68795667=80, srv2143047788=46, srv1405137645=10, srv369936074=62, srv761665995=86, srv1299753045=8, srv199582432=40, srv1433733687=11, srv202958649=42, srv2141602189=45, srv1711073204=23, srv192390416=34, srv98306469=98, srv1849868235=31, srv1033124062=1, srv1265940794=7, srv335385317=60, srv2085445741=43, srv700206220=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:46,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:46,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:46,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 11:53:46,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 11:53:46,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 11:53:46,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 11:53:46,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 11:53:46,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 11:53:46,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 11:53:46,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 11:53:46,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 11:53:46,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 11:53:46,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 11:53:46,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03795955474558878, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05774863289785541, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03795955474558878, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23608333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:46,117 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 11:53:46,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17181789219704485, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03795955474558878, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05774863289785541, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03795955474558878, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23608333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 11:53:46,326 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 297 ms to try 5000 different iterations. Found a solution that moves 2601 regions; Going from a computed imbalance of 0.17181789219704485 to a new imbalance of 0.014866321540736568. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023056721807345235); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03889610322693238, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.289, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023056721807345235); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:46,334 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1029669949:75 , srv1127882018:76 , srv1581717320:80 , srv2120472948:80 , srv1235591368:81 , srv1605798803:81 , srv1929812992:81 , srv202958649:81 , srv289521678:81 , srv307640114:81 , srv494822386:81 , srv638004106:81 , srv868053119:81 , srv1520502285:82 , srv1875014949:82 , srv1928909122:82 , srv357384470:82 , srv393556716:82 , srv667935927:82 , srv1228779221:83 , srv1711073204:83 , srv1743734641:83 , srv1792265557:83 , srv1807614208:83 , srv1921727002:83 , srv229899075:83 , srv239445402:83 , srv405324813:83 , srv1768119857:84 , srv2022671429:84 , srv2143047788:84 , srv254659623:84 , srv290812133:84 , srv606831325:84 , srv1042680835:85 , srv1535760983:85 , srv179645158:85 , srv199582432:85 , srv2141602189:85 , srv325225840:85 , srv369936074:85 , srv655865275:85 , srv68336041:85 , srv868771167:85 , srv93120499:85 , srv98306469:85 , srv1299753045:86 , srv1382749977:86 , srv1405137645:86 , srv1552513888:86 , srv2085445741:86 , srv223839949:86 , srv322922283:86 , srv469257556:86 , srv484251363:86 , srv599378060:86 , srv612204437:86 , srv67534236:86 , srv684994828:86 , srv68795667:86 , srv700206220:86 , srv758402839:86 , srv761665995:86 , srv773012907:86 , srv872338569:86 , srv1033124062:87 , srv1481052097:87 , srv1651539070:87 , srv1751982378:87 , srv1849868235:87 , srv192390416:87 , srv317512409:87 , srv514099285:87 , srv614511499:87 , srv961347036:87 , srv992704139:87 , srv1196329793:88 , srv1497057441:88 , srv1828075982:88 , srv1936553676:88 , srv1956425742:88 , srv335385317:88 , srv704017879:88 , srv77011820:88 , srv841972787:88 , srv941945316:88 , srv1471965241:89 , srv1950911721:89 , srv313950399:89 , srv690294933:89 , srv704213595:89 , srv1546650291:90 , srv294766880:90 , srv840828919:90 , srv1506994316:91 , srv928595276:91 , srv1265940794:92 , srv1433733687:92 , srv331772870:93 , srv673766745:552 } 2023-02-13 11:53:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1506994316=15, srv928595276=94, srv357384470=61, srv992704139=99, srv1928909122=35, srv1581717320=20, srv469257556=65, srv322922283=57, srv1875014949=32, srv655865275=74, srv872338569=93, srv1546650291=18, srv614511499=72, srv93120499=95, srv1936553676=37, srv606831325=70, srv1382749977=9, srv179645158=28, srv1651539070=22, srv1956425742=39, srv773012907=88, srv961347036=97, srv1950911721=38, srv667935927=75, srv1228779221=5, srv1471965241=12, srv1921727002=33, srv1929812992=36, srv1751982378=25, srv229899075=48, srv294766880=53, srv494822386=67, srv673766745=76, srv1497057441=14, srv840828919=89, srv325225840=58, srv1196329793=4, srv1605798803=21, srv1481052097=13, srv313950399=55, srv2022671429=41, srv868771167=92, srv1535760983=17, srv393556716=63, srv307640114=54, srv599378060=69, srv1807614208=29, srv684994828=79, srv1127882018=3, srv1743734641=24, srv77011820=87, srv941945316=96, srv1029669949=0, srv331772870=59, srv1828075982=30, srv484251363=66, srv758402839=85, srv2120472948=44, srv223839949=47, srv1552513888=19, srv289521678=51, srv68336041=78, srv1520502285=16, srv1792265557=27, srv690294933=81, srv841972787=90, srv1768119857=26, srv612204437=71, srv704213595=84, srv1235591368=6, srv254659623=50, srv67534236=77, srv317512409=56, srv1042680835=2, srv638004106=73, srv704017879=83, srv290812133=52, srv514099285=68, srv239445402=49, srv405324813=64, srv868053119=91, srv68795667=80, srv2143047788=46, srv1405137645=10, srv369936074=62, srv761665995=86, srv1299753045=8, srv199582432=40, srv1433733687=11, srv202958649=42, srv2141602189=45, srv1711073204=23, srv192390416=34, srv98306469=98, srv1849868235=31, srv1033124062=1, srv1265940794=7, srv335385317=60, srv2085445741=43, srv700206220=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:46,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:46,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:46,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:46,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 11:53:46,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 11:53:46,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 11:53:46,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 11:53:46,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 11:53:46,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 11:53:46,447 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023056721807345235); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03889610322693238, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023056721807345235); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:46,448 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 11:53:46,448 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014848108949471051, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023056721807345235); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03889610322693238, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023056721807345235); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 11:53:46,604 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 234 ms to try 5000 different iterations. Found a solution that moves 987 regions; Going from a computed imbalance of 0.014848108949471051 to a new imbalance of 7.457346318224246E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006442160669684254); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010572819481316787); MoveCostFunction : (multiplier=7.0, imbalance=0.10966666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006442160669684254); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:46,606 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1936553676:82 , srv405324813:82 , srv1228779221:83 , srv2143047788:83 , srv331772870:83 , srv369936074:83 , srv98306469:83 , srv1433733687:84 , srv1506994316:84 , srv1520502285:84 , srv1651539070:84 , srv2022671429:84 , srv229899075:84 , srv294766880:84 , srv325225840:84 , srv599378060:84 , srv612204437:84 , srv614511499:84 , srv68336041:84 , srv704213595:84 , srv93120499:84 , srv1042680835:85 , srv1265940794:85 , srv1382749977:85 , srv1405137645:85 , srv1471965241:85 , srv1552513888:85 , srv1605798803:85 , srv1768119857:85 , srv179645158:85 , srv1828075982:85 , srv2085445741:85 , srv2120472948:85 , srv2141602189:85 , srv223839949:85 , srv313950399:85 , srv322922283:85 , srv393556716:85 , srv469257556:85 , srv484251363:85 , srv655865275:85 , srv68795667:85 , srv690294933:85 , srv700206220:85 , srv761665995:85 , srv840828919:85 , srv841972787:85 , srv868771167:85 , srv928595276:85 , srv941945316:85 , srv992704139:85 , srv1033124062:86 , srv1196329793:86 , srv1299753045:86 , srv1497057441:86 , srv1546650291:86 , srv1711073204:86 , srv1807614208:86 , srv1921727002:86 , srv192390416:86 , srv254659623:86 , srv289521678:86 , srv307640114:86 , srv317512409:86 , srv335385317:86 , srv494822386:86 , srv514099285:86 , srv606831325:86 , srv67534236:86 , srv684994828:86 , srv773012907:86 , srv872338569:86 , srv961347036:86 , srv1029669949:87 , srv1127882018:87 , srv1235591368:87 , srv1481052097:87 , srv1535760983:87 , srv1581717320:87 , srv1743734641:87 , srv1751982378:87 , srv1875014949:87 , srv1928909122:87 , srv1950911721:87 , srv1956425742:87 , srv199582432:87 , srv202958649:87 , srv239445402:87 , srv290812133:87 , srv357384470:87 , srv638004106:87 , srv667935927:87 , srv704017879:87 , srv758402839:87 , srv77011820:87 , srv1849868235:88 , srv1929812992:88 , srv868053119:88 , srv1792265557:89 , srv673766745:536 } 2023-02-13 11:53:46,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:46,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1506994316=15, srv928595276=94, srv357384470=61, srv992704139=99, srv1928909122=35, srv1581717320=20, srv469257556=65, srv322922283=57, srv1875014949=32, srv655865275=74, srv872338569=93, srv1546650291=18, srv614511499=72, srv93120499=95, srv1936553676=37, srv606831325=70, srv1382749977=9, srv179645158=28, srv1651539070=22, srv1956425742=39, srv773012907=88, srv961347036=97, srv1950911721=38, srv667935927=75, srv1228779221=5, srv1471965241=12, srv1921727002=33, srv1929812992=36, srv1751982378=25, srv229899075=48, srv294766880=53, srv494822386=67, srv673766745=76, srv1497057441=14, srv840828919=89, srv325225840=58, srv1196329793=4, srv1605798803=21, srv1481052097=13, srv313950399=55, srv2022671429=41, srv868771167=92, srv1535760983=17, srv393556716=63, srv307640114=54, srv599378060=69, srv1807614208=29, srv684994828=79, srv1127882018=3, srv1743734641=24, srv77011820=87, srv941945316=96, srv1029669949=0, srv331772870=59, srv1828075982=30, srv484251363=66, srv758402839=85, srv2120472948=44, srv223839949=47, srv1552513888=19, srv289521678=51, srv68336041=78, srv1520502285=16, srv1792265557=27, srv690294933=81, srv841972787=90, srv1768119857=26, srv612204437=71, srv704213595=84, srv1235591368=6, srv254659623=50, srv67534236=77, srv317512409=56, srv1042680835=2, srv638004106=73, srv704017879=83, srv290812133=52, srv514099285=68, srv239445402=49, srv405324813=64, srv868053119=91, srv68795667=80, srv2143047788=46, srv1405137645=10, srv369936074=62, srv761665995=86, srv1299753045=8, srv199582432=40, srv1433733687=11, srv202958649=42, srv2141602189=45, srv1711073204=23, srv192390416=34, srv98306469=98, srv1849868235=31, srv1033124062=1, srv1265940794=7, srv335385317=60, srv2085445741=43, srv700206220=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:46,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:46,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:46,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:46,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:46,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:46,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 11:53:46,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 11:53:46,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 11:53:46,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 11:53:46,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 11:53:46,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 11:53:46,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006442160669684254); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010572819481316787); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006442160669684254); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:46,708 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 11:53:46,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.388235101080584E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006442160669684254); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010572819481316787); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006442160669684254); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 11:53:46,922 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 292 ms to try 5000 different iterations. Found a solution that moves 247 regions; Going from a computed imbalance of 7.388235101080584E-4 to a new imbalance of 3.464764599042148E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.027444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:46,923 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv704017879:82 , srv1196329793:83 , srv1581717320:83 , srv514099285:83 , srv599378060:83 , srv700206220:83 , srv1265940794:84 , srv1481052097:84 , srv1520502285:84 , srv1743734641:84 , srv1751982378:84 , srv179645158:84 , srv1849868235:84 , srv1875014949:84 , srv192390416:84 , srv1929812992:84 , srv1956425742:84 , srv199582432:84 , srv2022671429:84 , srv2085445741:84 , srv2141602189:84 , srv229899075:84 , srv290812133:84 , srv294766880:84 , srv307640114:84 , srv317512409:84 , srv322922283:84 , srv484251363:84 , srv494822386:84 , srv638004106:84 , srv67534236:84 , srv704213595:84 , srv77011820:84 , srv773012907:84 , srv840828919:84 , srv841972787:84 , srv868053119:84 , srv868771167:84 , srv872338569:84 , srv941945316:84 , srv992704139:84 , srv1029669949:85 , srv1033124062:85 , srv1042680835:85 , srv1127882018:85 , srv1235591368:85 , srv1299753045:85 , srv1382749977:85 , srv1405137645:85 , srv1433733687:85 , srv1471965241:85 , srv1497057441:85 , srv1506994316:85 , srv1535760983:85 , srv1546650291:85 , srv1552513888:85 , srv1605798803:85 , srv1711073204:85 , srv1768119857:85 , srv1792265557:85 , srv1828075982:85 , srv1928909122:85 , srv1936553676:85 , srv1950911721:85 , srv202958649:85 , srv2120472948:85 , srv2143047788:85 , srv223839949:85 , srv239445402:85 , srv289521678:85 , srv313950399:85 , srv325225840:85 , srv331772870:85 , srv335385317:85 , srv357384470:85 , srv369936074:85 , srv393556716:85 , srv405324813:85 , srv469257556:85 , srv606831325:85 , srv612204437:85 , srv614511499:85 , srv655865275:85 , srv667935927:85 , srv684994828:85 , srv68795667:85 , srv690294933:85 , srv758402839:85 , srv761665995:85 , srv928595276:85 , srv93120499:85 , srv961347036:85 , srv98306469:85 , srv1228779221:86 , srv1651539070:86 , srv1807614208:86 , srv1921727002:86 , srv254659623:86 , srv68336041:86 , srv673766745:627 } 2023-02-13 11:53:46,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:46,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1506994316=15, srv928595276=94, srv357384470=61, srv992704139=99, srv1928909122=35, srv1581717320=20, srv469257556=65, srv322922283=57, srv1875014949=32, srv655865275=74, srv872338569=93, srv1546650291=18, srv614511499=72, srv93120499=95, srv1936553676=37, srv606831325=70, srv1382749977=9, srv179645158=28, srv1651539070=22, srv1956425742=39, srv773012907=88, srv961347036=97, srv1950911721=38, srv667935927=75, srv1228779221=5, srv1471965241=12, srv1921727002=33, srv1929812992=36, srv1751982378=25, srv229899075=48, srv294766880=53, srv494822386=67, srv673766745=76, srv1497057441=14, srv840828919=89, srv325225840=58, srv1196329793=4, srv1605798803=21, srv1481052097=13, srv313950399=55, srv2022671429=41, srv868771167=92, srv1535760983=17, srv393556716=63, srv307640114=54, srv599378060=69, srv1807614208=29, srv684994828=79, srv1127882018=3, srv1743734641=24, srv77011820=87, srv941945316=96, srv1029669949=0, srv331772870=59, srv1828075982=30, srv484251363=66, srv758402839=85, srv2120472948=44, srv223839949=47, srv1552513888=19, srv289521678=51, srv68336041=78, srv1520502285=16, srv1792265557=27, srv690294933=81, srv841972787=90, srv1768119857=26, srv612204437=71, srv704213595=84, srv1235591368=6, srv254659623=50, srv67534236=77, srv317512409=56, srv1042680835=2, srv638004106=73, srv704017879=83, srv290812133=52, srv514099285=68, srv239445402=49, srv405324813=64, srv868053119=91, srv68795667=80, srv2143047788=46, srv1405137645=10, srv369936074=62, srv761665995=86, srv1299753045=8, srv199582432=40, srv1433733687=11, srv202958649=42, srv2141602189=45, srv1711073204=23, srv192390416=34, srv98306469=98, srv1849868235=31, srv1033124062=1, srv1265940794=7, srv335385317=60, srv2085445741=43, srv700206220=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:46,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:46,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:46,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:46,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:46,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:46,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:46,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 11:53:46,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 11:53:46,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 11:53:46,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 11:53:46,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 11:53:46,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 11:53:46,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 11:53:46,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 11:53:46,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 11:53:46,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 11:53:47,027 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:47,028 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 11:53:47,028 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.291811502441492E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 11:53:47,168 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 220 ms to try 5000 different iterations. Found a solution that moves 24 regions; Going from a computed imbalance of 3.291811502441492E-5 to a new imbalance of 6.912120645670642E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:47,169 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1433733687:83 , srv1792265557:83 , srv1921727002:83 , srv202958649:83 , srv313950399:83 , srv68336041:83 , srv761665995:83 , srv872338569:83 , srv1029669949:84 , srv1033124062:84 , srv1042680835:84 , srv1127882018:84 , srv1196329793:84 , srv1228779221:84 , srv1235591368:84 , srv1265940794:84 , srv1299753045:84 , srv1382749977:84 , srv1405137645:84 , srv1471965241:84 , srv1497057441:84 , srv1506994316:84 , srv1535760983:84 , srv1546650291:84 , srv1552513888:84 , srv1605798803:84 , srv1651539070:84 , srv1711073204:84 , srv1751982378:84 , srv1768119857:84 , srv1807614208:84 , srv1828075982:84 , srv1849868235:84 , srv1875014949:84 , srv192390416:84 , srv1928909122:84 , srv1929812992:84 , srv1936553676:84 , srv1950911721:84 , srv1956425742:84 , srv199582432:84 , srv2085445741:84 , srv2120472948:84 , srv2141602189:84 , srv2143047788:84 , srv223839949:84 , srv239445402:84 , srv254659623:84 , srv289521678:84 , srv290812133:84 , srv294766880:84 , srv307640114:84 , srv317512409:84 , srv322922283:84 , srv325225840:84 , srv331772870:84 , srv335385317:84 , srv357384470:84 , srv369936074:84 , srv393556716:84 , srv405324813:84 , srv469257556:84 , srv484251363:84 , srv494822386:84 , srv599378060:84 , srv606831325:84 , srv612204437:84 , srv614511499:84 , srv638004106:84 , srv655865275:84 , srv667935927:84 , srv67534236:84 , srv684994828:84 , srv68795667:84 , srv690294933:84 , srv700206220:84 , srv704017879:84 , srv704213595:84 , srv758402839:84 , srv77011820:84 , srv773012907:84 , srv840828919:84 , srv841972787:84 , srv868053119:84 , srv868771167:84 , srv928595276:84 , srv93120499:84 , srv941945316:84 , srv961347036:84 , srv98306469:84 , srv992704139:84 , srv1481052097:85 , srv1520502285:85 , srv1581717320:85 , srv1743734641:85 , srv179645158:85 , srv2022671429:85 , srv229899075:85 , srv514099285:85 , srv673766745:684 } 2023-02-13 11:53:47,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:47,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1506994316=15, srv928595276=94, srv357384470=61, srv992704139=99, srv1928909122=35, srv1581717320=20, srv469257556=65, srv322922283=57, srv1875014949=32, srv655865275=74, srv872338569=93, srv1546650291=18, srv614511499=72, srv93120499=95, srv1936553676=37, srv606831325=70, srv1382749977=9, srv179645158=28, srv1651539070=22, srv1956425742=39, srv773012907=88, srv961347036=97, srv1950911721=38, srv667935927=75, srv1228779221=5, srv1471965241=12, srv1921727002=33, srv1929812992=36, srv1751982378=25, srv229899075=48, srv294766880=53, srv494822386=67, srv673766745=76, srv1497057441=14, srv840828919=89, srv325225840=58, srv1196329793=4, srv1605798803=21, srv1481052097=13, srv313950399=55, srv2022671429=41, srv868771167=92, srv1535760983=17, srv393556716=63, srv307640114=54, srv599378060=69, srv1807614208=29, srv684994828=79, srv1127882018=3, srv1743734641=24, srv77011820=87, srv941945316=96, srv1029669949=0, srv331772870=59, srv1828075982=30, srv484251363=66, srv758402839=85, srv2120472948=44, srv223839949=47, srv1552513888=19, srv289521678=51, srv68336041=78, srv1520502285=16, srv1792265557=27, srv690294933=81, srv841972787=90, srv1768119857=26, srv612204437=71, srv704213595=84, srv1235591368=6, srv254659623=50, srv67534236=77, srv317512409=56, srv1042680835=2, srv638004106=73, srv704017879=83, srv290812133=52, srv514099285=68, srv239445402=49, srv405324813=64, srv868053119=91, srv68795667=80, srv2143047788=46, srv1405137645=10, srv369936074=62, srv761665995=86, srv1299753045=8, srv199582432=40, srv1433733687=11, srv202958649=42, srv2141602189=45, srv1711073204=23, srv192390416=34, srv98306469=98, srv1849868235=31, srv1033124062=1, srv1265940794=7, srv335385317=60, srv2085445741=43, srv700206220=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:47,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 11:53:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 11:53:47,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 11:53:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 11:53:47,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 11:53:47,270 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:47,271 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1433733687:83 , srv1792265557:83 , srv1921727002:83 , srv202958649:83 , srv313950399:83 , srv68336041:83 , srv761665995:83 , srv872338569:83 , srv1029669949:84 , srv1033124062:84 , srv1042680835:84 , srv1127882018:84 , srv1196329793:84 , srv1228779221:84 , srv1235591368:84 , srv1265940794:84 , srv1299753045:84 , srv1382749977:84 , srv1405137645:84 , srv1471965241:84 , srv1497057441:84 , srv1506994316:84 , srv1535760983:84 , srv1546650291:84 , srv1552513888:84 , srv1605798803:84 , srv1651539070:84 , srv1711073204:84 , srv1751982378:84 , srv1768119857:84 , srv1807614208:84 , srv1828075982:84 , srv1849868235:84 , srv1875014949:84 , srv192390416:84 , srv1928909122:84 , srv1929812992:84 , srv1936553676:84 , srv1950911721:84 , srv1956425742:84 , srv199582432:84 , srv2085445741:84 , srv2120472948:84 , srv2141602189:84 , srv2143047788:84 , srv223839949:84 , srv239445402:84 , srv254659623:84 , srv289521678:84 , srv290812133:84 , srv294766880:84 , srv307640114:84 , srv317512409:84 , srv322922283:84 , srv325225840:84 , srv331772870:84 , srv335385317:84 , srv357384470:84 , srv369936074:84 , srv393556716:84 , srv405324813:84 , srv469257556:84 , srv484251363:84 , srv494822386:84 , srv599378060:84 , srv606831325:84 , srv612204437:84 , srv614511499:84 , srv638004106:84 , srv655865275:84 , srv667935927:84 , srv67534236:84 , srv684994828:84 , srv68795667:84 , srv690294933:84 , srv700206220:84 , srv704017879:84 , srv704213595:84 , srv758402839:84 , srv77011820:84 , srv773012907:84 , srv840828919:84 , srv841972787:84 , srv868053119:84 , srv868771167:84 , srv928595276:84 , srv93120499:84 , srv941945316:84 , srv961347036:84 , srv98306469:84 , srv992704139:84 , srv1481052097:85 , srv1520502285:85 , srv1581717320:85 , srv1743734641:85 , srv179645158:85 , srv2022671429:85 , srv229899075:85 , srv514099285:85 , srv673766745:684 } 2023-02-13 11:53:47,299 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=301 (was 301), ProcessCount=169 (was 169), AvailableMemoryMB=5428 (was 5918) 2023-02-13 11:53:47,305 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=169, AvailableMemoryMB=5427 2023-02-13 11:53:47,306 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 11:53:47,306 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-13 11:53:47,315 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037500712:84 , srv1208084400:84 , srv1216248425:84 , srv1272498926:84 , srv1285976521:84 , srv1573236714:84 , srv159494685:84 , srv1634701959:84 , srv168666070:84 , srv1688475422:84 , srv1725955506:84 , srv1854599161:84 , srv1872695718:84 , srv1954511334:84 , srv2030834660:84 , srv210549797:84 , srv231131670:84 , srv255788127:84 , srv344501284:84 , srv386229436:84 , srv511859566:84 , srv513273021:84 , srv613372382:84 , srv667308602:84 , srv739873592:84 , srv761297298:84 , srv921433311:84 , srv960137428:84 , srv966407192:84 , srv1230775248:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-13 11:53:47,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:53:47,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv739873592=25, srv168666070=9, srv2030834660=15, srv1573236714=6, srv921433311=27, srv1854599161=12, srv960137428=28, srv761297298=26, srv1208084400=1, srv231131670=17, srv966407192=29, srv1954511334=14, srv667308602=24, srv344501284=19, srv210549797=16, srv1272498926=4, srv1688475422=10, srv1872695718=13, srv613372382=23, srv1285976521=5, srv159494685=7, srv1725955506=11, srv513273021=22, srv1634701959=8, srv255788127=18, srv1230775248=3, srv386229436=20, srv1037500712=0, srv511859566=21, srv1216248425=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:53:47,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:53:47,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:53:47,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-13 11:53:47,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1523029107268825, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:53:47,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912672944762831, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1523029107268825, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-13 11:55:47,330 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9769610 different iterations. Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912672944762831 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:55:47,337 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037500712:90 , srv1208084400:90 , srv1216248425:90 , srv1230775248:90 , srv1272498926:90 , srv1285976521:90 , srv1573236714:90 , srv159494685:90 , srv1634701959:90 , srv168666070:90 , srv1688475422:90 , srv1725955506:90 , srv1854599161:90 , srv1872695718:90 , srv1954511334:90 , srv2030834660:90 , srv210549797:90 , srv231131670:90 , srv255788127:90 , srv344501284:90 , srv386229436:90 , srv511859566:90 , srv513273021:90 , srv613372382:90 , srv667308602:90 , srv739873592:90 , srv761297298:90 , srv921433311:90 , srv960137428:90 , srv966407192:90 } 2023-02-13 11:55:47,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 11:55:47,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv739873592=25, srv168666070=9, srv2030834660=15, srv1573236714=6, srv921433311=27, srv1854599161=12, srv960137428=28, srv761297298=26, srv1208084400=1, srv231131670=17, srv966407192=29, srv1954511334=14, srv667308602=24, srv344501284=19, srv210549797=16, srv1272498926=4, srv1688475422=10, srv1872695718=13, srv613372382=23, srv1285976521=5, srv159494685=7, srv1725955506=11, srv513273021=22, srv1634701959=8, srv255788127=18, srv1230775248=3, srv386229436=20, srv1037500712=0, srv511859566=21, srv1216248425=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 11:55:47,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 11:55:47,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 11:55:47,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 11:55:47,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 11:55:47,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 11:55:47,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 11:55:47,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-13 11:55:47,357 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 11:55:47,357 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1037500712:90 , srv1208084400:90 , srv1216248425:90 , srv1230775248:90 , srv1272498926:90 , srv1285976521:90 , srv1573236714:90 , srv159494685:90 , srv1634701959:90 , srv168666070:90 , srv1688475422:90 , srv1725955506:90 , srv1854599161:90 , srv1872695718:90 , srv1954511334:90 , srv2030834660:90 , srv210549797:90 , srv231131670:90 , srv255788127:90 , srv344501284:90 , srv386229436:90 , srv511859566:90 , srv513273021:90 , srv613372382:90 , srv667308602:90 , srv739873592:90 , srv761297298:90 , srv921433311:90 , srv960137428:90 , srv966407192:90 } 2023-02-13 11:55:47,373 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=182 (was 301), ProcessCount=168 (was 169), AvailableMemoryMB=5310 (was 5427)