2023-03-17 20:53:00,846 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-17 20:53:01,166 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-17 20:53:01,268 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-17 20:53:01,328 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-17 20:53:01,334 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-17 20:53:01,346 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=169, AvailableMemoryMB=3315 2023-03-17 20:53:01,348 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-17 20:53:01,348 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-17 20:53:01,473 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv10031210:84 , srv1011685713:84 , srv1021872783:84 , srv1054963052:84 , srv1070380097:84 , srv1075476493:84 , srv1111841953:84 , srv1120085684:84 , srv1149273753:84 , srv1162300037:84 , srv1215479421:84 , srv1218394911:84 , srv1240608391:84 , srv1266408715:84 , srv1269021850:84 , srv1282123510:84 , srv1315516727:84 , srv1323423460:84 , srv1328319976:84 , srv1335939261:84 , srv1390828948:84 , srv1411243316:84 , srv1435108900:84 , srv1457331054:84 , srv1570257814:84 , srv1574332870:84 , srv1586815075:84 , srv1593439209:84 , srv1597168748:84 , srv1651159661:84 , srv1666321722:84 , srv172699868:84 , srv1739463802:84 , srv174106032:84 , srv1783812227:84 , srv1824847756:84 , srv1853801955:84 , srv1861592603:84 , srv1873372120:84 , srv1878361405:84 , srv1899083278:84 , srv195553349:84 , srv1972086651:84 , srv1987737922:84 , srv2007301819:84 , srv2008749538:84 , srv201031538:84 , srv2072847273:84 , srv2077834453:84 , srv2085401350:84 , srv2089924859:84 , srv271103437:84 , srv281334817:84 , srv308445227:84 , srv319487311:84 , srv332174659:84 , srv337372496:84 , srv345923402:84 , srv403573249:84 , srv415723172:84 , srv435417275:84 , srv438918956:84 , srv442994964:84 , srv458331507:84 , srv470944842:84 , srv476285683:84 , srv487535776:84 , srv490953311:84 , srv505533230:84 , srv515707739:84 , srv517582679:84 , srv531891457:84 , srv53404389:84 , srv540986563:84 , srv584044029:84 , srv601551755:84 , srv605819415:84 , srv613613996:84 , srv660167185:84 , srv660696126:84 , srv663523510:84 , srv682771591:84 , srv702914417:84 , srv78529647:84 , srv809619831:84 , srv836444450:84 , srv840063321:84 , srv847044987:84 , srv848520516:84 , srv855711895:84 , srv88336053:84 , srv920303971:84 , srv944774728:84 , srv951478575:84 , srv957662845:84 , srv960376699:84 , srv966381400:84 , srv978137300:84 , srv991881673:84 , srv221126639:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-17 20:53:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:01,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1111841953=6, srv1282123510=15, srv487535776=67, srv702914417=83, srv1853801955=36, srv951478575=94, srv1149273753=8, srv505533230=69, srv840063321=87, srv660167185=79, srv660696126=80, srv1075476493=5, srv847044987=88, srv613613996=78, srv1021872783=2, srv960376699=96, srv308445227=54, srv1335939261=19, srv1666321722=30, srv1824847756=35, srv271103437=52, srv2077834453=48, srv345923402=58, srv281334817=53, srv1328319976=18, srv1972086651=42, srv2089924859=50, srv458331507=64, srv836444450=86, srv1323423460=17, srv172699868=31, srv957662845=95, srv991881673=99, srv10031210=0, srv337372496=57, srv1861592603=37, srv195553349=41, srv490953311=68, srv584044029=75, srv2072847273=47, srv605819415=77, srv848520516=89, srv1070380097=4, srv1215479421=10, srv174106032=33, srv855711895=90, srv1240608391=12, srv221126639=51, srv435417275=61, srv1457331054=23, srv1570257814=24, srv1651159661=29, srv1597168748=28, srv809619831=85, srv1411243316=21, srv415723172=60, srv476285683=66, srv1783812227=34, srv1218394911=11, srv1269021850=14, srv78529647=84, srv438918956=62, srv682771591=82, srv1873372120=38, srv1162300037=9, srv1593439209=27, srv2085401350=49, srv201031538=46, srv53404389=73, srv1390828948=20, srv442994964=63, srv1435108900=22, srv88336053=91, srv2007301819=44, srv319487311=55, srv2008749538=45, srv403573249=59, srv1266408715=13, srv663523510=81, srv1987737922=43, srv332174659=56, srv1011685713=1, srv1120085684=7, srv944774728=93, srv978137300=98, srv1586815075=26, srv1315516727=16, srv966381400=97, srv920303971=92, srv1574332870=25, srv517582679=71, srv1899083278=40, srv531891457=72, srv515707739=70, srv601551755=76, srv1054963052=3, srv540986563=74, srv1878361405=39, srv470944842=65, srv1739463802=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:01,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:01,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:01,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:01,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:01,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:01,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:01,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:01,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-17 20:53:01,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-17 20:53:01,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-17 20:53:01,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-17 20:53:01,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-17 20:53:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-17 20:53:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-17 20:53:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-17 20:53:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-17 20:53:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-17 20:53:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-17 20:53:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-17 20:53:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-17 20:53:01,756 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:01,757 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-17 20:53:01,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-17 20:53:02,082 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 478 ms to try 5000 different iterations. Found a solution that moves 4634 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18949344899809545. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04278874464054425, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0587352675130679, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5148888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04278874464054425, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:02,096 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1070380097:81 , srv1266408715:81 , srv1651159661:81 , srv1783812227:81 , srv271103437:81 , srv308445227:81 , srv1861592603:82 , srv195553349:82 , srv951478575:82 , srv1075476493:83 , srv1149273753:83 , srv2077834453:83 , srv319487311:83 , srv345923402:83 , srv438918956:83 , srv442994964:83 , srv470944842:83 , srv836444450:83 , srv855711895:83 , srv960376699:83 , srv978137300:83 , srv1120085684:84 , srv1218394911:84 , srv1328319976:84 , srv201031538:84 , srv490953311:84 , srv682771591:84 , srv840063321:84 , srv1011685713:85 , srv1021872783:85 , srv1111841953:85 , srv1457331054:85 , srv1586815075:85 , srv1597168748:85 , srv1739463802:85 , srv2089924859:85 , srv332174659:85 , srv337372496:85 , srv458331507:85 , srv476285683:85 , srv515707739:85 , srv531891457:85 , srv540986563:85 , srv660696126:85 , srv848520516:85 , srv920303971:85 , srv944774728:85 , srv1215479421:86 , srv1323423460:86 , srv1335939261:86 , srv1853801955:86 , srv1987737922:86 , srv2008749538:86 , srv415723172:86 , srv605819415:86 , srv613613996:86 , srv663523510:86 , srv966381400:86 , srv1054963052:87 , srv1162300037:87 , srv1240608391:87 , srv1390828948:87 , srv1411243316:87 , srv1435108900:87 , srv1593439209:87 , srv1666321722:87 , srv1873372120:87 , srv1972086651:87 , srv2007301819:87 , srv281334817:87 , srv403573249:87 , srv584044029:87 , srv702914417:87 , srv1282123510:88 , srv1570257814:88 , srv601551755:88 , srv78529647:88 , srv809619831:88 , srv957662845:88 , srv10031210:89 , srv2085401350:89 , srv435417275:89 , srv487535776:89 , srv505533230:89 , srv53404389:89 , srv172699868:90 , srv1899083278:90 , srv517582679:90 , srv1824847756:91 , srv1878361405:91 , srv660167185:91 , srv847044987:91 , srv991881673:91 , srv2072847273:92 , srv1315516727:93 , srv88336053:93 , srv1269021850:94 , srv174106032:94 , srv1574332870:95 , srv221126639:470 } 2023-03-17 20:53:02,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:02,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1111841953=6, srv1282123510=15, srv487535776=67, srv702914417=83, srv1853801955=36, srv951478575=94, srv1149273753=8, srv505533230=69, srv840063321=87, srv660167185=79, srv660696126=80, srv1075476493=5, srv847044987=88, srv613613996=78, srv1021872783=2, srv960376699=96, srv308445227=54, srv1335939261=19, srv1666321722=30, srv1824847756=35, srv271103437=52, srv2077834453=48, srv345923402=58, srv281334817=53, srv1328319976=18, srv1972086651=42, srv2089924859=50, srv458331507=64, srv836444450=86, srv1323423460=17, srv172699868=31, srv957662845=95, srv991881673=99, srv10031210=0, srv337372496=57, srv1861592603=37, srv195553349=41, srv490953311=68, srv584044029=75, srv2072847273=47, srv605819415=77, srv848520516=89, srv1070380097=4, srv1215479421=10, srv174106032=33, srv855711895=90, srv1240608391=12, srv221126639=51, srv435417275=61, srv1457331054=23, srv1570257814=24, srv1651159661=29, srv1597168748=28, srv809619831=85, srv1411243316=21, srv415723172=60, srv476285683=66, srv1783812227=34, srv1218394911=11, srv1269021850=14, srv78529647=84, srv438918956=62, srv682771591=82, srv1873372120=38, srv1162300037=9, srv1593439209=27, srv2085401350=49, srv201031538=46, srv53404389=73, srv1390828948=20, srv442994964=63, srv1435108900=22, srv88336053=91, srv2007301819=44, srv319487311=55, srv2008749538=45, srv403573249=59, srv1266408715=13, srv663523510=81, srv1987737922=43, srv332174659=56, srv1011685713=1, srv1120085684=7, srv944774728=93, srv978137300=98, srv1586815075=26, srv1315516727=16, srv966381400=97, srv920303971=92, srv1574332870=25, srv517582679=71, srv1899083278=40, srv531891457=72, srv515707739=70, srv601551755=76, srv1054963052=3, srv540986563=74, srv1878361405=39, srv470944842=65, srv1739463802=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:02,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:02,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:02,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:02,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:02,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:02,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:02,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-17 20:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-17 20:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-17 20:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-17 20:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-17 20:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-17 20:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-17 20:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-17 20:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-17 20:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-17 20:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-17 20:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-17 20:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-17 20:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-17 20:53:02,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-17 20:53:02,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-17 20:53:02,253 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04278874464054425, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0587352675130679, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04278874464054425, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:02,253 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-17 20:53:02,253 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18946100103657124, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04278874464054425, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0587352675130679, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04278874464054425, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-17 20:53:02,431 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 261 ms to try 5000 different iterations. Found a solution that moves 2677 regions; Going from a computed imbalance of 0.18946100103657124 to a new imbalance of 0.015396452117140674. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02363257425474367); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03593234695647119, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29744444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02363257425474367); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:02,441 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1574332870:77 , srv1315516727:78 , srv174106032:78 , srv1824847756:80 , srv1899083278:80 , srv88336053:80 , srv1269021850:81 , srv660167185:81 , srv10031210:82 , srv2072847273:82 , srv2085401350:82 , srv403573249:82 , srv1282123510:83 , srv1435108900:83 , srv1570257814:83 , srv2007301819:83 , srv435417275:83 , srv487535776:83 , srv505533230:83 , srv517582679:83 , srv809619831:83 , srv847044987:83 , srv991881673:83 , srv1240608391:84 , srv1593439209:84 , srv172699868:84 , srv2008749538:84 , srv601551755:84 , srv702914417:84 , srv944774728:84 , srv957662845:84 , srv1011685713:85 , srv1021872783:85 , srv1054963052:85 , srv1218394911:85 , srv1323423460:85 , srv1390828948:85 , srv1972086651:85 , srv1987737922:85 , srv476285683:85 , srv53404389:85 , srv605819415:85 , srv660696126:85 , srv848520516:85 , srv966381400:85 , srv1586815075:86 , srv1597168748:86 , srv1666321722:86 , srv1878361405:86 , srv201031538:86 , srv2089924859:86 , srv281334817:86 , srv332174659:86 , srv337372496:86 , srv458331507:86 , srv531891457:86 , srv613613996:86 , srv663523510:86 , srv682771591:86 , srv78529647:86 , srv1111841953:87 , srv1120085684:87 , srv1162300037:87 , srv1215479421:87 , srv1328319976:87 , srv1335939261:87 , srv1411243316:87 , srv1853801955:87 , srv1873372120:87 , srv319487311:87 , srv540986563:87 , srv836444450:87 , srv840063321:87 , srv920303971:87 , srv1457331054:88 , srv1739463802:88 , srv2077834453:88 , srv345923402:88 , srv855711895:88 , srv195553349:89 , srv271103437:89 , srv415723172:89 , srv438918956:89 , srv442994964:89 , srv584044029:89 , srv951478575:89 , srv960376699:89 , srv978137300:89 , srv1075476493:90 , srv1149273753:90 , srv1861592603:90 , srv470944842:90 , srv515707739:90 , srv1070380097:91 , srv1651159661:91 , srv1783812227:91 , srv490953311:91 , srv1266408715:92 , srv308445227:93 , srv221126639:514 } 2023-03-17 20:53:02,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:02,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1111841953=6, srv1282123510=15, srv487535776=67, srv702914417=83, srv1853801955=36, srv951478575=94, srv1149273753=8, srv505533230=69, srv840063321=87, srv660167185=79, srv660696126=80, srv1075476493=5, srv847044987=88, srv613613996=78, srv1021872783=2, srv960376699=96, srv308445227=54, srv1335939261=19, srv1666321722=30, srv1824847756=35, srv271103437=52, srv2077834453=48, srv345923402=58, srv281334817=53, srv1328319976=18, srv1972086651=42, srv2089924859=50, srv458331507=64, srv836444450=86, srv1323423460=17, srv172699868=31, srv957662845=95, srv991881673=99, srv10031210=0, srv337372496=57, srv1861592603=37, srv195553349=41, srv490953311=68, srv584044029=75, srv2072847273=47, srv605819415=77, srv848520516=89, srv1070380097=4, srv1215479421=10, srv174106032=33, srv855711895=90, srv1240608391=12, srv221126639=51, srv435417275=61, srv1457331054=23, srv1570257814=24, srv1651159661=29, srv1597168748=28, srv809619831=85, srv1411243316=21, srv415723172=60, srv476285683=66, srv1783812227=34, srv1218394911=11, srv1269021850=14, srv78529647=84, srv438918956=62, srv682771591=82, srv1873372120=38, srv1162300037=9, srv1593439209=27, srv2085401350=49, srv201031538=46, srv53404389=73, srv1390828948=20, srv442994964=63, srv1435108900=22, srv88336053=91, srv2007301819=44, srv319487311=55, srv2008749538=45, srv403573249=59, srv1266408715=13, srv663523510=81, srv1987737922=43, srv332174659=56, srv1011685713=1, srv1120085684=7, srv944774728=93, srv978137300=98, srv1586815075=26, srv1315516727=16, srv966381400=97, srv920303971=92, srv1574332870=25, srv517582679=71, srv1899083278=40, srv531891457=72, srv515707739=70, srv601551755=76, srv1054963052=3, srv540986563=74, srv1878361405=39, srv470944842=65, srv1739463802=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-17 20:53:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-17 20:53:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-17 20:53:02,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-17 20:53:02,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-17 20:53:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-17 20:53:02,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-17 20:53:02,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-17 20:53:02,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-17 20:53:02,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-17 20:53:02,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-17 20:53:02,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-17 20:53:02,553 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02363257425474367); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03593234695647119, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02363257425474367); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:02,553 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-17 20:53:02,553 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015377707362500999, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02363257425474367); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03593234695647119, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02363257425474367); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-17 20:53:02,742 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 262 ms to try 5000 different iterations. Found a solution that moves 1084 regions; Going from a computed imbalance of 0.015377707362500999 to a new imbalance of 3.9186005910585043E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004895720153351054); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626566); MoveCostFunction : (multiplier=7.0, imbalance=0.12044444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004895720153351054); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:02,744 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv584044029:81 , srv1878361405:82 , srv1075476493:83 , srv1269021850:83 , srv2072847273:83 , srv415723172:83 , srv470944842:83 , srv490953311:83 , srv515707739:83 , srv53404389:83 , srv78529647:83 , srv1162300037:84 , srv1411243316:84 , srv1457331054:84 , srv1666321722:84 , srv172699868:84 , srv1873372120:84 , srv281334817:84 , srv308445227:84 , srv517582679:84 , srv847044987:84 , srv991881673:84 , srv1054963052:85 , srv1070380097:85 , srv1111841953:85 , srv1149273753:85 , srv1215479421:85 , srv1266408715:85 , srv1390828948:85 , srv1651159661:85 , srv1739463802:85 , srv1972086651:85 , srv1987737922:85 , srv435417275:85 , srv442994964:85 , srv487535776:85 , srv601551755:85 , srv702914417:85 , srv88336053:85 , srv960376699:85 , srv978137300:85 , srv10031210:86 , srv1011685713:86 , srv1021872783:86 , srv1240608391:86 , srv1315516727:86 , srv1323423460:86 , srv1335939261:86 , srv1570257814:86 , srv1574332870:86 , srv1593439209:86 , srv1597168748:86 , srv1783812227:86 , srv1853801955:86 , srv1861592603:86 , srv2077834453:86 , srv2089924859:86 , srv438918956:86 , srv505533230:86 , srv531891457:86 , srv540986563:86 , srv613613996:86 , srv660167185:86 , srv663523510:86 , srv809619831:86 , srv855711895:86 , srv920303971:86 , srv951478575:86 , srv957662845:86 , srv966381400:86 , srv1120085684:87 , srv1282123510:87 , srv1435108900:87 , srv1586815075:87 , srv174106032:87 , srv1899083278:87 , srv195553349:87 , srv2085401350:87 , srv337372496:87 , srv345923402:87 , srv458331507:87 , srv605819415:87 , srv836444450:87 , srv840063321:87 , srv848520516:87 , srv1824847756:88 , srv2007301819:88 , srv2008749538:88 , srv201031538:88 , srv271103437:88 , srv319487311:88 , srv332174659:88 , srv476285683:88 , srv660696126:88 , srv682771591:88 , srv944774728:88 , srv1218394911:89 , srv1328319976:89 , srv403573249:89 , srv221126639:517 } 2023-03-17 20:53:02,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:02,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1111841953=6, srv1282123510=15, srv487535776=67, srv702914417=83, srv1853801955=36, srv951478575=94, srv1149273753=8, srv505533230=69, srv840063321=87, srv660167185=79, srv660696126=80, srv1075476493=5, srv847044987=88, srv613613996=78, srv1021872783=2, srv960376699=96, srv308445227=54, srv1335939261=19, srv1666321722=30, srv1824847756=35, srv271103437=52, srv2077834453=48, srv345923402=58, srv281334817=53, srv1328319976=18, srv1972086651=42, srv2089924859=50, srv458331507=64, srv836444450=86, srv1323423460=17, srv172699868=31, srv957662845=95, srv991881673=99, srv10031210=0, srv337372496=57, srv1861592603=37, srv195553349=41, srv490953311=68, srv584044029=75, srv2072847273=47, srv605819415=77, srv848520516=89, srv1070380097=4, srv1215479421=10, srv174106032=33, srv855711895=90, srv1240608391=12, srv221126639=51, srv435417275=61, srv1457331054=23, srv1570257814=24, srv1651159661=29, srv1597168748=28, srv809619831=85, srv1411243316=21, srv415723172=60, srv476285683=66, srv1783812227=34, srv1218394911=11, srv1269021850=14, srv78529647=84, srv438918956=62, srv682771591=82, srv1873372120=38, srv1162300037=9, srv1593439209=27, srv2085401350=49, srv201031538=46, srv53404389=73, srv1390828948=20, srv442994964=63, srv1435108900=22, srv88336053=91, srv2007301819=44, srv319487311=55, srv2008749538=45, srv403573249=59, srv1266408715=13, srv663523510=81, srv1987737922=43, srv332174659=56, srv1011685713=1, srv1120085684=7, srv944774728=93, srv978137300=98, srv1586815075=26, srv1315516727=16, srv966381400=97, srv920303971=92, srv1574332870=25, srv517582679=71, srv1899083278=40, srv531891457=72, srv515707739=70, srv601551755=76, srv1054963052=3, srv540986563=74, srv1878361405=39, srv470944842=65, srv1739463802=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:02,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:02,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-17 20:53:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-17 20:53:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-17 20:53:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-17 20:53:02,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-17 20:53:02,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-17 20:53:02,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-17 20:53:02,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-17 20:53:02,853 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004895720153351054); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626566); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004895720153351054); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:02,853 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-17 20:53:02,853 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.8426972887446937E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004895720153351054); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626566); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004895720153351054); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-17 20:53:03,147 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 372 ms to try 5000 different iterations. Found a solution that moves 281 regions; Going from a computed imbalance of 3.8426972887446937E-4 to a new imbalance of 3.025207969467734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.03122222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:03,151 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1328319976:82 , srv1282123510:83 , srv1335939261:83 , srv1574332870:83 , srv172699868:83 , srv174106032:83 , srv1824847756:83 , srv1853801955:83 , srv1878361405:83 , srv319487311:83 , srv332174659:83 , srv1120085684:84 , srv1149273753:84 , srv1162300037:84 , srv1215479421:84 , srv1218394911:84 , srv1269021850:84 , srv1323423460:84 , srv1390828948:84 , srv1411243316:84 , srv1586815075:84 , srv1593439209:84 , srv1783812227:84 , srv1861592603:84 , srv1873372120:84 , srv195553349:84 , srv2007301819:84 , srv2008749538:84 , srv2085401350:84 , srv271103437:84 , srv308445227:84 , srv337372496:84 , srv345923402:84 , srv403573249:84 , srv415723172:84 , srv438918956:84 , srv458331507:84 , srv490953311:84 , srv505533230:84 , srv515707739:84 , srv531891457:84 , srv605819415:84 , srv613613996:84 , srv660696126:84 , srv663523510:84 , srv682771591:84 , srv840063321:84 , srv847044987:84 , srv88336053:84 , srv920303971:84 , srv957662845:84 , srv991881673:84 , srv10031210:85 , srv1054963052:85 , srv1070380097:85 , srv1111841953:85 , srv1240608391:85 , srv1266408715:85 , srv1315516727:85 , srv1435108900:85 , srv1457331054:85 , srv1570257814:85 , srv1651159661:85 , srv1666321722:85 , srv1739463802:85 , srv1899083278:85 , srv1972086651:85 , srv1987737922:85 , srv201031538:85 , srv2077834453:85 , srv2089924859:85 , srv442994964:85 , srv476285683:85 , srv487535776:85 , srv517582679:85 , srv53404389:85 , srv540986563:85 , srv584044029:85 , srv601551755:85 , srv660167185:85 , srv702914417:85 , srv78529647:85 , srv809619831:85 , srv836444450:85 , srv848520516:85 , srv855711895:85 , srv944774728:85 , srv951478575:85 , srv960376699:85 , srv966381400:85 , srv978137300:85 , srv1011685713:86 , srv1021872783:86 , srv1075476493:86 , srv1597168748:86 , srv2072847273:86 , srv281334817:86 , srv435417275:86 , srv470944842:86 , srv221126639:641 } 2023-03-17 20:53:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1111841953=6, srv1282123510=15, srv487535776=67, srv702914417=83, srv1853801955=36, srv951478575=94, srv1149273753=8, srv505533230=69, srv840063321=87, srv660167185=79, srv660696126=80, srv1075476493=5, srv847044987=88, srv613613996=78, srv1021872783=2, srv960376699=96, srv308445227=54, srv1335939261=19, srv1666321722=30, srv1824847756=35, srv271103437=52, srv2077834453=48, srv345923402=58, srv281334817=53, srv1328319976=18, srv1972086651=42, srv2089924859=50, srv458331507=64, srv836444450=86, srv1323423460=17, srv172699868=31, srv957662845=95, srv991881673=99, srv10031210=0, srv337372496=57, srv1861592603=37, srv195553349=41, srv490953311=68, srv584044029=75, srv2072847273=47, srv605819415=77, srv848520516=89, srv1070380097=4, srv1215479421=10, srv174106032=33, srv855711895=90, srv1240608391=12, srv221126639=51, srv435417275=61, srv1457331054=23, srv1570257814=24, srv1651159661=29, srv1597168748=28, srv809619831=85, srv1411243316=21, srv415723172=60, srv476285683=66, srv1783812227=34, srv1218394911=11, srv1269021850=14, srv78529647=84, srv438918956=62, srv682771591=82, srv1873372120=38, srv1162300037=9, srv1593439209=27, srv2085401350=49, srv201031538=46, srv53404389=73, srv1390828948=20, srv442994964=63, srv1435108900=22, srv88336053=91, srv2007301819=44, srv319487311=55, srv2008749538=45, srv403573249=59, srv1266408715=13, srv663523510=81, srv1987737922=43, srv332174659=56, srv1011685713=1, srv1120085684=7, srv944774728=93, srv978137300=98, srv1586815075=26, srv1315516727=16, srv966381400=97, srv920303971=92, srv1574332870=25, srv517582679=71, srv1899083278=40, srv531891457=72, srv515707739=70, srv601551755=76, srv1054963052=3, srv540986563=74, srv1878361405=39, srv470944842=65, srv1739463802=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:03,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:03,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:03,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:03,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-17 20:53:03,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-17 20:53:03,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-17 20:53:03,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-17 20:53:03,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-17 20:53:03,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-17 20:53:03,256 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:03,256 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-17 20:53:03,256 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.8284475640232626E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-17 20:53:03,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 231 ms to try 5000 different iterations. Found a solution that moves 36 regions; Going from a computed imbalance of 2.8284475640232626E-5 to a new imbalance of 7.639834888070601E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.004); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:03,414 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1266408715:83 , srv1597168748:83 , srv1739463802:83 , srv201031538:83 , srv2072847273:83 , srv281334817:83 , srv435417275:83 , srv476285683:83 , srv540986563:83 , srv660167185:83 , srv10031210:84 , srv1011685713:84 , srv1021872783:84 , srv1054963052:84 , srv1070380097:84 , srv1075476493:84 , srv1111841953:84 , srv1120085684:84 , srv1149273753:84 , srv1162300037:84 , srv1215479421:84 , srv1218394911:84 , srv1240608391:84 , srv1269021850:84 , srv1315516727:84 , srv1328319976:84 , srv1335939261:84 , srv1411243316:84 , srv1435108900:84 , srv1457331054:84 , srv1570257814:84 , srv1586815075:84 , srv1651159661:84 , srv1666321722:84 , srv174106032:84 , srv1783812227:84 , srv1824847756:84 , srv1853801955:84 , srv1861592603:84 , srv1873372120:84 , srv1878361405:84 , srv1899083278:84 , srv195553349:84 , srv1972086651:84 , srv2007301819:84 , srv2008749538:84 , srv2077834453:84 , srv2085401350:84 , srv2089924859:84 , srv271103437:84 , srv308445227:84 , srv332174659:84 , srv337372496:84 , srv345923402:84 , srv403573249:84 , srv415723172:84 , srv438918956:84 , srv442994964:84 , srv458331507:84 , srv470944842:84 , srv487535776:84 , srv490953311:84 , srv505533230:84 , srv515707739:84 , srv517582679:84 , srv53404389:84 , srv584044029:84 , srv601551755:84 , srv605819415:84 , srv613613996:84 , srv660696126:84 , srv663523510:84 , srv682771591:84 , srv78529647:84 , srv809619831:84 , srv836444450:84 , srv840063321:84 , srv847044987:84 , srv848520516:84 , srv855711895:84 , srv88336053:84 , srv920303971:84 , srv944774728:84 , srv951478575:84 , srv957662845:84 , srv960376699:84 , srv966381400:84 , srv978137300:84 , srv991881673:84 , srv1282123510:85 , srv1323423460:85 , srv1390828948:85 , srv1574332870:85 , srv1593439209:85 , srv172699868:85 , srv1987737922:85 , srv319487311:85 , srv531891457:85 , srv702914417:85 , srv221126639:684 } 2023-03-17 20:53:03,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:03,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1111841953=6, srv1282123510=15, srv487535776=67, srv702914417=83, srv1853801955=36, srv951478575=94, srv1149273753=8, srv505533230=69, srv840063321=87, srv660167185=79, srv660696126=80, srv1075476493=5, srv847044987=88, srv613613996=78, srv1021872783=2, srv960376699=96, srv308445227=54, srv1335939261=19, srv1666321722=30, srv1824847756=35, srv271103437=52, srv2077834453=48, srv345923402=58, srv281334817=53, srv1328319976=18, srv1972086651=42, srv2089924859=50, srv458331507=64, srv836444450=86, srv1323423460=17, srv172699868=31, srv957662845=95, srv991881673=99, srv10031210=0, srv337372496=57, srv1861592603=37, srv195553349=41, srv490953311=68, srv584044029=75, srv2072847273=47, srv605819415=77, srv848520516=89, srv1070380097=4, srv1215479421=10, srv174106032=33, srv855711895=90, srv1240608391=12, srv221126639=51, srv435417275=61, srv1457331054=23, srv1570257814=24, srv1651159661=29, srv1597168748=28, srv809619831=85, srv1411243316=21, srv415723172=60, srv476285683=66, srv1783812227=34, srv1218394911=11, srv1269021850=14, srv78529647=84, srv438918956=62, srv682771591=82, srv1873372120=38, srv1162300037=9, srv1593439209=27, srv2085401350=49, srv201031538=46, srv53404389=73, srv1390828948=20, srv442994964=63, srv1435108900=22, srv88336053=91, srv2007301819=44, srv319487311=55, srv2008749538=45, srv403573249=59, srv1266408715=13, srv663523510=81, srv1987737922=43, srv332174659=56, srv1011685713=1, srv1120085684=7, srv944774728=93, srv978137300=98, srv1586815075=26, srv1315516727=16, srv966381400=97, srv920303971=92, srv1574332870=25, srv517582679=71, srv1899083278=40, srv531891457=72, srv515707739=70, srv601551755=76, srv1054963052=3, srv540986563=74, srv1878361405=39, srv470944842=65, srv1739463802=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:03,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:03,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:03,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:03,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:03,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-17 20:53:03,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-17 20:53:03,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-17 20:53:03,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-17 20:53:03,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-17 20:53:03,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-17 20:53:03,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-17 20:53:03,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-17 20:53:03,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-17 20:53:03,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-17 20:53:03,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-17 20:53:03,544 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:03,544 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1266408715:83 , srv1597168748:83 , srv1739463802:83 , srv201031538:83 , srv2072847273:83 , srv281334817:83 , srv435417275:83 , srv476285683:83 , srv540986563:83 , srv660167185:83 , srv10031210:84 , srv1011685713:84 , srv1021872783:84 , srv1054963052:84 , srv1070380097:84 , srv1075476493:84 , srv1111841953:84 , srv1120085684:84 , srv1149273753:84 , srv1162300037:84 , srv1215479421:84 , srv1218394911:84 , srv1240608391:84 , srv1269021850:84 , srv1315516727:84 , srv1328319976:84 , srv1335939261:84 , srv1411243316:84 , srv1435108900:84 , srv1457331054:84 , srv1570257814:84 , srv1586815075:84 , srv1651159661:84 , srv1666321722:84 , srv174106032:84 , srv1783812227:84 , srv1824847756:84 , srv1853801955:84 , srv1861592603:84 , srv1873372120:84 , srv1878361405:84 , srv1899083278:84 , srv195553349:84 , srv1972086651:84 , srv2007301819:84 , srv2008749538:84 , srv2077834453:84 , srv2085401350:84 , srv2089924859:84 , srv271103437:84 , srv308445227:84 , srv332174659:84 , srv337372496:84 , srv345923402:84 , srv403573249:84 , srv415723172:84 , srv438918956:84 , srv442994964:84 , srv458331507:84 , srv470944842:84 , srv487535776:84 , srv490953311:84 , srv505533230:84 , srv515707739:84 , srv517582679:84 , srv53404389:84 , srv584044029:84 , srv601551755:84 , srv605819415:84 , srv613613996:84 , srv660696126:84 , srv663523510:84 , srv682771591:84 , srv78529647:84 , srv809619831:84 , srv836444450:84 , srv840063321:84 , srv847044987:84 , srv848520516:84 , srv855711895:84 , srv88336053:84 , srv920303971:84 , srv944774728:84 , srv951478575:84 , srv957662845:84 , srv960376699:84 , srv966381400:84 , srv978137300:84 , srv991881673:84 , srv1282123510:85 , srv1323423460:85 , srv1390828948:85 , srv1574332870:85 , srv1593439209:85 , srv172699868:85 , srv1987737922:85 , srv319487311:85 , srv531891457:85 , srv702914417:85 , srv221126639:684 } 2023-03-17 20:53:03,570 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 334), ProcessCount=169 (was 169), AvailableMemoryMB=2696 (was 3315) 2023-03-17 20:53:03,579 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=169, AvailableMemoryMB=2695 2023-03-17 20:53:03,580 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-17 20:53:03,580 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-17 20:53:03,591 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004657638:84 , srv1082775996:84 , srv1084756937:84 , srv12531226:84 , srv1354131800:84 , srv1373986163:84 , srv1401624452:84 , srv1513684143:84 , srv1673593461:84 , srv1762523548:84 , srv178869264:84 , srv1841290237:84 , srv1867956552:84 , srv1940214396:84 , srv2116482563:84 , srv282896318:84 , srv296919786:84 , srv349043189:84 , srv379587136:84 , srv523417755:84 , srv532700796:84 , srv638340615:84 , srv700965005:84 , srv721634993:84 , srv786172553:84 , srv815622250:84 , srv873012382:84 , srv94742268:84 , srv966498743:84 , srv635832562:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-17 20:53:03,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:53:03,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv94742268=28, srv1940214396=13, srv1082775996=1, srv1841290237=11, srv296919786=16, srv1373986163=5, srv635832562=21, srv873012382=27, srv523417755=19, srv282896318=15, srv1513684143=7, srv966498743=29, srv786172553=25, srv815622250=26, srv178869264=10, srv532700796=20, srv1401624452=6, srv638340615=22, srv1867956552=12, srv721634993=24, srv1004657638=0, srv700965005=23, srv1673593461=8, srv12531226=3, srv1762523548=9, srv1354131800=4, srv2116482563=14, srv349043189=17, srv379587136=18, srv1084756937=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:53:03,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:53:03,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:53:03,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:53:03,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:53:03,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:53:03,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:53:03,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:53:03,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-17 20:53:03,636 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1701391514372776, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:53:03,637 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912729146174902, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1701391514372776, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-17 20:55:03,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9342429 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912729146174902 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:55:03,624 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004657638:90 , srv1082775996:90 , srv1084756937:90 , srv12531226:90 , srv1354131800:90 , srv1373986163:90 , srv1401624452:90 , srv1513684143:90 , srv1673593461:90 , srv1762523548:90 , srv178869264:90 , srv1841290237:90 , srv1867956552:90 , srv1940214396:90 , srv2116482563:90 , srv282896318:90 , srv296919786:90 , srv349043189:90 , srv379587136:90 , srv523417755:90 , srv532700796:90 , srv635832562:90 , srv638340615:90 , srv700965005:90 , srv721634993:90 , srv786172553:90 , srv815622250:90 , srv873012382:90 , srv94742268:90 , srv966498743:90 } 2023-03-17 20:55:03,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-17 20:55:03,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv94742268=28, srv1940214396=13, srv1082775996=1, srv1841290237=11, srv296919786=16, srv1373986163=5, srv635832562=21, srv873012382=27, srv523417755=19, srv282896318=15, srv1513684143=7, srv966498743=29, srv786172553=25, srv815622250=26, srv178869264=10, srv532700796=20, srv1401624452=6, srv638340615=22, srv1867956552=12, srv721634993=24, srv1004657638=0, srv700965005=23, srv1673593461=8, srv12531226=3, srv1762523548=9, srv1354131800=4, srv2116482563=14, srv349043189=17, srv379587136=18, srv1084756937=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-17 20:55:03,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-17 20:55:03,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-17 20:55:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-17 20:55:03,640 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-17 20:55:03,640 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1004657638:90 , srv1082775996:90 , srv1084756937:90 , srv12531226:90 , srv1354131800:90 , srv1373986163:90 , srv1401624452:90 , srv1513684143:90 , srv1673593461:90 , srv1762523548:90 , srv178869264:90 , srv1841290237:90 , srv1867956552:90 , srv1940214396:90 , srv2116482563:90 , srv282896318:90 , srv296919786:90 , srv349043189:90 , srv379587136:90 , srv523417755:90 , srv532700796:90 , srv635832562:90 , srv638340615:90 , srv700965005:90 , srv721634993:90 , srv786172553:90 , srv815622250:90 , srv873012382:90 , srv94742268:90 , srv966498743:90 } 2023-03-17 20:55:03,650 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 334), ProcessCount=177 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=1846 (was 2695)