2024-04-07 11:53:15,513 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-07 11:53:15,829 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-07 11:53:15,937 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-07 11:53:15,998 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-07 11:53:16,005 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-07 11:53:16,021 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=346, ProcessCount=171, AvailableMemoryMB=7453 2024-04-07 11:53:16,023 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-07 11:53:16,023 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-07 11:53:16,160 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003522253:84 , srv1016668458:84 , srv1020825465:84 , srv103044595:84 , srv103426226:84 , srv1065115920:84 , srv1098742374:84 , srv1120346573:84 , srv1123587450:84 , srv1124883874:84 , srv1135533112:84 , srv1149218026:84 , srv1182941913:84 , srv1188230381:84 , srv1192994323:84 , srv1193389815:84 , srv1198526285:84 , srv1212940832:84 , srv121954190:84 , srv1220030384:84 , srv1258959704:84 , srv1292086771:84 , srv1297931594:84 , srv1302126054:84 , srv1311693799:84 , srv1317200194:84 , srv1321547570:84 , srv1329143850:84 , srv1335289828:84 , srv1361114204:84 , srv1418359096:84 , srv1425149658:84 , srv1428815295:84 , srv1442075312:84 , srv1449835519:84 , srv1466164083:84 , srv1467237953:84 , srv1483884137:84 , srv1544100146:84 , srv1551422092:84 , srv15787304:84 , srv1612018790:84 , srv1628561085:84 , srv1764968718:84 , srv1785492637:84 , srv1817695757:84 , srv1818631405:84 , srv182564192:84 , srv1834367081:84 , srv1850574473:84 , srv1854897827:84 , srv1859157695:84 , srv1862055737:84 , srv1864415740:84 , srv1866746757:84 , srv1907863725:84 , srv1919152937:84 , srv1967537971:84 , srv197921918:84 , srv1979798452:84 , srv1993973936:84 , srv2032371974:84 , srv2041257999:84 , srv2083561613:84 , srv2097211903:84 , srv2107063290:84 , srv2133709109:84 , srv248511581:84 , srv295866794:84 , srv362115625:84 , srv380059345:84 , srv393159585:84 , srv418284352:84 , srv419539316:84 , srv421313952:84 , srv425833235:84 , srv42771850:84 , srv443533973:84 , srv475265531:84 , srv526390088:84 , srv549487426:84 , srv588404093:84 , srv615958063:84 , srv644028:84 , srv647982473:84 , srv653549515:84 , srv690256121:84 , srv705276968:84 , srv734916047:84 , srv738699357:84 , srv768200392:84 , srv835699459:84 , srv835748444:84 , srv83965199:84 , srv840050903:84 , srv853702667:84 , srv904546334:84 , srv971506885:84 , srv982937166:84 , srv1819888605:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-07 11:53:16,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1818631405=46, srv1864415740=54, srv1442075312=33, srv644028=84, srv647982473=85, srv1065115920=5, srv1425149658=31, srv835748444=93, srv853702667=96, srv1220030384=19, srv1428815295=32, srv425833235=76, srv1124883874=9, srv103426226=4, srv1329143850=27, srv1764968718=43, srv1466164083=35, srv835699459=92, srv1628561085=42, srv1321547570=26, srv1123587450=8, srv1016668458=1, srv1612018790=41, srv1188230381=13, srv1907863725=56, srv393159585=72, srv1919152937=57, srv904546334=97, srv1335289828=28, srv1785492637=44, srv1182941913=12, srv653549515=86, srv1817695757=45, srv1979798452=60, srv1834367081=49, srv1198526285=16, srv182564192=48, srv690256121=87, srv121954190=18, srv705276968=88, srv248511581=68, srv475265531=79, srv1258959704=20, srv1993973936=61, srv734916047=89, srv1850574473=50, srv2133709109=67, srv2083561613=64, srv1020825465=2, srv2032371974=62, srv83965199=94, srv1120346573=7, srv42771850=77, srv2107063290=66, srv1483884137=37, srv971506885=98, srv738699357=90, srv1819888605=47, srv1859157695=52, srv1551422092=39, srv1467237953=36, srv15787304=40, srv1193389815=15, srv295866794=69, srv362115625=70, srv840050903=95, srv1317200194=25, srv1212940832=17, srv1449835519=34, srv1003522253=0, srv1135533112=10, srv103044595=3, srv1854897827=51, srv2097211903=65, srv1149218026=11, srv419539316=74, srv421313952=75, srv526390088=80, srv2041257999=63, srv1967537971=58, srv1544100146=38, srv380059345=71, srv1418359096=30, srv1866746757=55, srv1297931594=22, srv549487426=81, srv1311693799=24, srv768200392=91, srv1098742374=6, srv443533973=78, srv1862055737=53, srv1292086771=21, srv197921918=59, srv1302126054=23, srv615958063=83, srv588404093=82, srv982937166=99, srv1192994323=14, srv1361114204=29, srv418284352=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:16,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:16,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:16,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:16,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:16,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:16,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:16,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:16,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:16,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:16,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:16,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:16,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:16,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:16,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-07 11:53:16,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-07 11:53:16,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-07 11:53:16,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-07 11:53:16,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-07 11:53:16,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-07 11:53:16,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-07 11:53:16,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-07 11:53:16,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-07 11:53:16,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-07 11:53:16,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-07 11:53:16,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-07 11:53:16,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-07 11:53:16,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-07 11:53:16,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-07 11:53:16,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-07 11:53:16,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-07 11:53:16,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-07 11:53:16,499 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:16,500 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-07 11:53:16,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-07 11:53:16,983 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 680 ms to try 5000 different iterations. Found a solution that moves 4689 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17856526247141213. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03714160323154887, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051951803616331726, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.521, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03714160323154887, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.246, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:16,999 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv295866794:82 , srv1329143850:83 , srv1418359096:83 , srv1551422092:83 , srv2107063290:83 , srv2133709109:83 , srv647982473:83 , srv738699357:83 , srv1212940832:84 , srv1425149658:84 , srv1449835519:84 , srv1834367081:84 , srv1864415740:84 , srv1979798452:84 , srv2032371974:84 , srv615958063:84 , srv103426226:85 , srv1123587450:85 , srv1149218026:85 , srv1297931594:85 , srv1311693799:85 , srv1321547570:85 , srv1335289828:85 , srv1466164083:85 , srv1467237953:85 , srv15787304:85 , srv1628561085:85 , srv1859157695:85 , srv1862055737:85 , srv380059345:85 , srv475265531:85 , srv588404093:85 , srv653549515:85 , srv734916047:85 , srv840050903:85 , srv1020825465:86 , srv103044595:86 , srv1135533112:86 , srv1258959704:86 , srv1428815295:86 , srv1442075312:86 , srv1612018790:86 , srv1817695757:86 , srv1818631405:86 , srv2041257999:86 , srv418284352:86 , srv421313952:86 , srv690256121:86 , srv904546334:86 , srv971506885:86 , srv1016668458:87 , srv1065115920:87 , srv1124883874:87 , srv1188230381:87 , srv1192994323:87 , srv1292086771:87 , srv1302126054:87 , srv1483884137:87 , srv1544100146:87 , srv1919152937:87 , srv197921918:87 , srv1993973936:87 , srv2097211903:87 , srv362115625:87 , srv42771850:87 , srv549487426:87 , srv644028:87 , srv705276968:87 , srv835748444:87 , srv83965199:87 , srv982937166:87 , srv1003522253:88 , srv1193389815:88 , srv1198526285:88 , srv1220030384:88 , srv1907863725:88 , srv1967537971:88 , srv2083561613:88 , srv393159585:88 , srv419539316:88 , srv526390088:88 , srv853702667:88 , srv1098742374:89 , srv1317200194:89 , srv1361114204:89 , srv248511581:89 , srv443533973:89 , srv768200392:89 , srv835699459:89 , srv1785492637:90 , srv121954190:91 , srv1850574473:91 , srv1120346573:92 , srv1182941913:92 , srv1764968718:92 , srv425833235:92 , srv182564192:93 , srv1854897827:93 , srv1866746757:96 , srv1819888605:420 } 2024-04-07 11:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1818631405=46, srv1864415740=54, srv1442075312=33, srv644028=84, srv647982473=85, srv1065115920=5, srv1425149658=31, srv835748444=93, srv853702667=96, srv1220030384=19, srv1428815295=32, srv425833235=76, srv1124883874=9, srv103426226=4, srv1329143850=27, srv1764968718=43, srv1466164083=35, srv835699459=92, srv1628561085=42, srv1321547570=26, srv1123587450=8, srv1016668458=1, srv1612018790=41, srv1188230381=13, srv1907863725=56, srv393159585=72, srv1919152937=57, srv904546334=97, srv1335289828=28, srv1785492637=44, srv1182941913=12, srv653549515=86, srv1817695757=45, srv1979798452=60, srv1834367081=49, srv1198526285=16, srv182564192=48, srv690256121=87, srv121954190=18, srv705276968=88, srv248511581=68, srv475265531=79, srv1258959704=20, srv1993973936=61, srv734916047=89, srv1850574473=50, srv2133709109=67, srv2083561613=64, srv1020825465=2, srv2032371974=62, srv83965199=94, srv1120346573=7, srv42771850=77, srv2107063290=66, srv1483884137=37, srv971506885=98, srv738699357=90, srv1819888605=47, srv1859157695=52, srv1551422092=39, srv1467237953=36, srv15787304=40, srv1193389815=15, srv295866794=69, srv362115625=70, srv840050903=95, srv1317200194=25, srv1212940832=17, srv1449835519=34, srv1003522253=0, srv1135533112=10, srv103044595=3, srv1854897827=51, srv2097211903=65, srv1149218026=11, srv419539316=74, srv421313952=75, srv526390088=80, srv2041257999=63, srv1967537971=58, srv1544100146=38, srv380059345=71, srv1418359096=30, srv1866746757=55, srv1297931594=22, srv549487426=81, srv1311693799=24, srv768200392=91, srv1098742374=6, srv443533973=78, srv1862055737=53, srv1292086771=21, srv197921918=59, srv1302126054=23, srv615958063=83, srv588404093=82, srv982937166=99, srv1192994323=14, srv1361114204=29, srv418284352=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:17,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:17,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:17,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:17,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:17,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-07 11:53:17,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-07 11:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-07 11:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-07 11:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-07 11:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-07 11:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-07 11:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-07 11:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-07 11:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-07 11:53:17,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03714160323154887, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051951803616331726, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03714160323154887, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.246, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:17,259 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-07 11:53:17,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17853242939165664, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03714160323154887, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051951803616331726, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03714160323154887, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.246, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-07 11:53:17,558 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 485 ms to try 5000 different iterations. Found a solution that moves 2683 regions; Going from a computed imbalance of 0.17853242939165664 to a new imbalance of 0.015210985754362543. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019592430288355382); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02904387331056087, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2981111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019592430288355382); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:17,570 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1866746757:76 , srv1854897827:79 , srv1182941913:80 , srv121954190:81 , srv1317200194:82 , srv182564192:82 , srv1850574473:82 , srv248511581:82 , srv425833235:82 , srv1120346573:83 , srv1764968718:83 , srv1907863725:83 , srv1967537971:83 , srv419539316:83 , srv526390088:83 , srv768200392:83 , srv835699459:83 , srv103044595:84 , srv1098742374:84 , srv1220030384:84 , srv1361114204:84 , srv1544100146:84 , srv1785492637:84 , srv362115625:84 , srv421313952:84 , srv42771850:84 , srv443533973:84 , srv549487426:84 , srv835748444:84 , srv83965199:84 , srv1003522253:85 , srv1135533112:85 , srv1188230381:85 , srv1192994323:85 , srv1193389815:85 , srv1198526285:85 , srv1212940832:85 , srv1297931594:85 , srv1428815295:85 , srv1442075312:85 , srv1466164083:85 , srv15787304:85 , srv1817695757:85 , srv1818631405:85 , srv197921918:85 , srv2083561613:85 , srv393159585:85 , srv418284352:85 , srv690256121:85 , srv840050903:85 , srv1020825465:86 , srv103426226:86 , srv1065115920:86 , srv1124883874:86 , srv1292086771:86 , srv1302126054:86 , srv1612018790:86 , srv2097211903:86 , srv475265531:86 , srv644028:86 , srv853702667:86 , srv982937166:86 , srv1016668458:87 , srv1123587450:87 , srv1149218026:87 , srv1311693799:87 , srv1335289828:87 , srv1425149658:87 , srv1483884137:87 , srv1551422092:87 , srv1628561085:87 , srv1859157695:87 , srv1862055737:87 , srv1919152937:87 , srv1993973936:87 , srv2133709109:87 , srv380059345:87 , srv705276968:87 , srv904546334:87 , srv1329143850:88 , srv1418359096:88 , srv1834367081:88 , srv2041257999:88 , srv653549515:88 , srv734916047:88 , srv738699357:88 , srv971506885:88 , srv1449835519:89 , srv1467237953:89 , srv1864415740:89 , srv1979798452:89 , srv615958063:89 , srv647982473:89 , srv1258959704:90 , srv2032371974:90 , srv2107063290:90 , srv295866794:90 , srv588404093:90 , srv1321547570:91 , srv1819888605:528 } 2024-04-07 11:53:17,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:17,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1818631405=46, srv1864415740=54, srv1442075312=33, srv644028=84, srv647982473=85, srv1065115920=5, srv1425149658=31, srv835748444=93, srv853702667=96, srv1220030384=19, srv1428815295=32, srv425833235=76, srv1124883874=9, srv103426226=4, srv1329143850=27, srv1764968718=43, srv1466164083=35, srv835699459=92, srv1628561085=42, srv1321547570=26, srv1123587450=8, srv1016668458=1, srv1612018790=41, srv1188230381=13, srv1907863725=56, srv393159585=72, srv1919152937=57, srv904546334=97, srv1335289828=28, srv1785492637=44, srv1182941913=12, srv653549515=86, srv1817695757=45, srv1979798452=60, srv1834367081=49, srv1198526285=16, srv182564192=48, srv690256121=87, srv121954190=18, srv705276968=88, srv248511581=68, srv475265531=79, srv1258959704=20, srv1993973936=61, srv734916047=89, srv1850574473=50, srv2133709109=67, srv2083561613=64, srv1020825465=2, srv2032371974=62, srv83965199=94, srv1120346573=7, srv42771850=77, srv2107063290=66, srv1483884137=37, srv971506885=98, srv738699357=90, srv1819888605=47, srv1859157695=52, srv1551422092=39, srv1467237953=36, srv15787304=40, srv1193389815=15, srv295866794=69, srv362115625=70, srv840050903=95, srv1317200194=25, srv1212940832=17, srv1449835519=34, srv1003522253=0, srv1135533112=10, srv103044595=3, srv1854897827=51, srv2097211903=65, srv1149218026=11, srv419539316=74, srv421313952=75, srv526390088=80, srv2041257999=63, srv1967537971=58, srv1544100146=38, srv380059345=71, srv1418359096=30, srv1866746757=55, srv1297931594=22, srv549487426=81, srv1311693799=24, srv768200392=91, srv1098742374=6, srv443533973=78, srv1862055737=53, srv1292086771=21, srv197921918=59, srv1302126054=23, srv615958063=83, srv588404093=82, srv982937166=99, srv1192994323=14, srv1361114204=29, srv418284352=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:17,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-07 11:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-07 11:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-07 11:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-07 11:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-07 11:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-07 11:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-07 11:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-07 11:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-07 11:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-07 11:53:17,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019592430288355382); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02904387331056087, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019592430288355382); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:17,763 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-07 11:53:17,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015192198986824909, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019592430288355382); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02904387331056087, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019592430288355382); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-07 11:53:18,177 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 546 ms to try 5000 different iterations. Found a solution that moves 1058 regions; Going from a computed imbalance of 0.015192198986824909 to a new imbalance of 6.400727520172169E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00424936634728175); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007668789048930225); MoveCostFunction : (multiplier=7.0, imbalance=0.11755555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00424936634728175); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:18,180 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1258959704:82 , srv1321547570:82 , srv588404093:82 , srv1065115920:83 , srv1120346573:83 , srv1467237953:83 , srv1764968718:83 , srv182564192:83 , srv1993973936:83 , srv615958063:83 , srv705276968:83 , srv1016668458:84 , srv1123587450:84 , srv1124883874:84 , srv1193389815:84 , srv1198526285:84 , srv1292086771:84 , srv1361114204:84 , srv1785492637:84 , srv1866746757:84 , srv2032371974:84 , srv2097211903:84 , srv425833235:84 , srv644028:84 , srv835699459:84 , srv853702667:84 , srv971506885:84 , srv982937166:84 , srv1003522253:85 , srv1098742374:85 , srv1149218026:85 , srv1192994323:85 , srv1220030384:85 , srv1302126054:85 , srv1311693799:85 , srv1449835519:85 , srv1483884137:85 , srv1850574473:85 , srv1864415740:85 , srv1919152937:85 , srv197921918:85 , srv1979798452:85 , srv2041257999:85 , srv2083561613:85 , srv2107063290:85 , srv295866794:85 , srv393159585:85 , srv443533973:85 , srv647982473:85 , srv653549515:85 , srv734916047:85 , srv768200392:85 , srv904546334:85 , srv1020825465:86 , srv1135533112:86 , srv1182941913:86 , srv1188230381:86 , srv121954190:86 , srv1317200194:86 , srv1329143850:86 , srv1335289828:86 , srv1428815295:86 , srv1544100146:86 , srv1817695757:86 , srv1854897827:86 , srv1859157695:86 , srv1862055737:86 , srv380059345:86 , srv418284352:86 , srv42771850:86 , srv475265531:86 , srv738699357:86 , srv835748444:86 , srv103044595:87 , srv1297931594:87 , srv1418359096:87 , srv1425149658:87 , srv1442075312:87 , srv1466164083:87 , srv1551422092:87 , srv15787304:87 , srv1612018790:87 , srv1818631405:87 , srv1907863725:87 , srv248511581:87 , srv362115625:87 , srv419539316:87 , srv421313952:87 , srv526390088:87 , srv549487426:87 , srv83965199:87 , srv840050903:87 , srv103426226:88 , srv1212940832:88 , srv1834367081:88 , srv1967537971:88 , srv2133709109:88 , srv690256121:88 , srv1628561085:89 , srv1819888605:547 } 2024-04-07 11:53:18,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:18,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1818631405=46, srv1864415740=54, srv1442075312=33, srv644028=84, srv647982473=85, srv1065115920=5, srv1425149658=31, srv835748444=93, srv853702667=96, srv1220030384=19, srv1428815295=32, srv425833235=76, srv1124883874=9, srv103426226=4, srv1329143850=27, srv1764968718=43, srv1466164083=35, srv835699459=92, srv1628561085=42, srv1321547570=26, srv1123587450=8, srv1016668458=1, srv1612018790=41, srv1188230381=13, srv1907863725=56, srv393159585=72, srv1919152937=57, srv904546334=97, srv1335289828=28, srv1785492637=44, srv1182941913=12, srv653549515=86, srv1817695757=45, srv1979798452=60, srv1834367081=49, srv1198526285=16, srv182564192=48, srv690256121=87, srv121954190=18, srv705276968=88, srv248511581=68, srv475265531=79, srv1258959704=20, srv1993973936=61, srv734916047=89, srv1850574473=50, srv2133709109=67, srv2083561613=64, srv1020825465=2, srv2032371974=62, srv83965199=94, srv1120346573=7, srv42771850=77, srv2107063290=66, srv1483884137=37, srv971506885=98, srv738699357=90, srv1819888605=47, srv1859157695=52, srv1551422092=39, srv1467237953=36, srv15787304=40, srv1193389815=15, srv295866794=69, srv362115625=70, srv840050903=95, srv1317200194=25, srv1212940832=17, srv1449835519=34, srv1003522253=0, srv1135533112=10, srv103044595=3, srv1854897827=51, srv2097211903=65, srv1149218026=11, srv419539316=74, srv421313952=75, srv526390088=80, srv2041257999=63, srv1967537971=58, srv1544100146=38, srv380059345=71, srv1418359096=30, srv1866746757=55, srv1297931594=22, srv549487426=81, srv1311693799=24, srv768200392=91, srv1098742374=6, srv443533973=78, srv1862055737=53, srv1292086771=21, srv197921918=59, srv1302126054=23, srv615958063=83, srv588404093=82, srv982937166=99, srv1192994323=14, srv1361114204=29, srv418284352=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:18,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:18,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:18,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-07 11:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-07 11:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-07 11:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-07 11:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-07 11:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-07 11:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-07 11:53:18,346 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00424936634728175); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007668789048930225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00424936634728175); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:18,346 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-07 11:53:18,347 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.326644776769944E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00424936634728175); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007668789048930225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00424936634728175); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-07 11:53:18,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 432 ms to try 5000 different iterations. Found a solution that moves 219 regions; Going from a computed imbalance of 6.326644776769944E-4 to a new imbalance of 2.611532459657771E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.024333333333333332); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:18,656 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1628561085:81 , srv1834367081:82 , srv103426226:83 , srv1425149658:83 , srv1483884137:83 , srv1612018790:83 , srv1919152937:83 , srv1967537971:83 , srv2041257999:83 , srv690256121:83 , srv1003522253:84 , srv1020825465:84 , srv1098742374:84 , srv1120346573:84 , srv1149218026:84 , srv1182941913:84 , srv1188230381:84 , srv121954190:84 , srv1258959704:84 , srv1302126054:84 , srv1321547570:84 , srv1335289828:84 , srv1442075312:84 , srv1449835519:84 , srv1764968718:84 , srv1785492637:84 , srv1818631405:84 , srv182564192:84 , srv1850574473:84 , srv1854897827:84 , srv1859157695:84 , srv1862055737:84 , srv1864415740:84 , srv1907863725:84 , srv1979798452:84 , srv2032371974:84 , srv2083561613:84 , srv2107063290:84 , srv2133709109:84 , srv248511581:84 , srv362115625:84 , srv380059345:84 , srv393159585:84 , srv419539316:84 , srv425833235:84 , srv526390088:84 , srv549487426:84 , srv653549515:84 , srv734916047:84 , srv83965199:84 , srv853702667:84 , srv904546334:84 , srv971506885:84 , srv1016668458:85 , srv103044595:85 , srv1124883874:85 , srv1135533112:85 , srv1192994323:85 , srv1193389815:85 , srv1198526285:85 , srv1212940832:85 , srv1220030384:85 , srv1292086771:85 , srv1297931594:85 , srv1311693799:85 , srv1317200194:85 , srv1329143850:85 , srv1361114204:85 , srv1418359096:85 , srv1428815295:85 , srv1466164083:85 , srv1467237953:85 , srv1544100146:85 , srv1551422092:85 , srv15787304:85 , srv1817695757:85 , srv1866746757:85 , srv197921918:85 , srv1993973936:85 , srv2097211903:85 , srv295866794:85 , srv418284352:85 , srv421313952:85 , srv42771850:85 , srv443533973:85 , srv475265531:85 , srv588404093:85 , srv644028:85 , srv647982473:85 , srv705276968:85 , srv738699357:85 , srv768200392:85 , srv835748444:85 , srv840050903:85 , srv982937166:85 , srv1065115920:86 , srv1123587450:86 , srv615958063:86 , srv835699459:86 , srv1819888605:647 } 2024-04-07 11:53:18,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:18,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1818631405=46, srv1864415740=54, srv1442075312=33, srv644028=84, srv647982473=85, srv1065115920=5, srv1425149658=31, srv835748444=93, srv853702667=96, srv1220030384=19, srv1428815295=32, srv425833235=76, srv1124883874=9, srv103426226=4, srv1329143850=27, srv1764968718=43, srv1466164083=35, srv835699459=92, srv1628561085=42, srv1321547570=26, srv1123587450=8, srv1016668458=1, srv1612018790=41, srv1188230381=13, srv1907863725=56, srv393159585=72, srv1919152937=57, srv904546334=97, srv1335289828=28, srv1785492637=44, srv1182941913=12, srv653549515=86, srv1817695757=45, srv1979798452=60, srv1834367081=49, srv1198526285=16, srv182564192=48, srv690256121=87, srv121954190=18, srv705276968=88, srv248511581=68, srv475265531=79, srv1258959704=20, srv1993973936=61, srv734916047=89, srv1850574473=50, srv2133709109=67, srv2083561613=64, srv1020825465=2, srv2032371974=62, srv83965199=94, srv1120346573=7, srv42771850=77, srv2107063290=66, srv1483884137=37, srv971506885=98, srv738699357=90, srv1819888605=47, srv1859157695=52, srv1551422092=39, srv1467237953=36, srv15787304=40, srv1193389815=15, srv295866794=69, srv362115625=70, srv840050903=95, srv1317200194=25, srv1212940832=17, srv1449835519=34, srv1003522253=0, srv1135533112=10, srv103044595=3, srv1854897827=51, srv2097211903=65, srv1149218026=11, srv419539316=74, srv421313952=75, srv526390088=80, srv2041257999=63, srv1967537971=58, srv1544100146=38, srv380059345=71, srv1418359096=30, srv1866746757=55, srv1297931594=22, srv549487426=81, srv1311693799=24, srv768200392=91, srv1098742374=6, srv443533973=78, srv1862055737=53, srv1292086771=21, srv197921918=59, srv1302126054=23, srv615958063=83, srv588404093=82, srv982937166=99, srv1192994323=14, srv1361114204=29, srv418284352=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-07 11:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-07 11:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-07 11:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-07 11:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-07 11:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-07 11:53:18,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:18,780 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-07 11:53:18,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.458185382104963E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-07 11:53:19,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 335 ms to try 5000 different iterations. Found a solution that moves 19 regions; Going from a computed imbalance of 2.458185382104963E-5 to a new imbalance of 6.8771098973709144E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.002111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:19,018 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1016668458:83 , srv1418359096:83 , srv443533973:83 , srv1003522253:84 , srv1020825465:84 , srv103044595:84 , srv103426226:84 , srv1065115920:84 , srv1098742374:84 , srv1120346573:84 , srv1123587450:84 , srv1124883874:84 , srv1135533112:84 , srv1182941913:84 , srv1188230381:84 , srv1192994323:84 , srv1193389815:84 , srv1198526285:84 , srv1212940832:84 , srv121954190:84 , srv1220030384:84 , srv1258959704:84 , srv1292086771:84 , srv1297931594:84 , srv1302126054:84 , srv1311693799:84 , srv1317200194:84 , srv1321547570:84 , srv1329143850:84 , srv1335289828:84 , srv1361114204:84 , srv1428815295:84 , srv1442075312:84 , srv1449835519:84 , srv1466164083:84 , srv1467237953:84 , srv1483884137:84 , srv1544100146:84 , srv1551422092:84 , srv15787304:84 , srv1612018790:84 , srv1628561085:84 , srv1764968718:84 , srv1785492637:84 , srv1817695757:84 , srv1818631405:84 , srv182564192:84 , srv1834367081:84 , srv1850574473:84 , srv1854897827:84 , srv1859157695:84 , srv1862055737:84 , srv1864415740:84 , srv1907863725:84 , srv1919152937:84 , srv1967537971:84 , srv197921918:84 , srv1979798452:84 , srv1993973936:84 , srv2032371974:84 , srv2041257999:84 , srv2083561613:84 , srv2097211903:84 , srv2107063290:84 , srv2133709109:84 , srv248511581:84 , srv295866794:84 , srv362115625:84 , srv380059345:84 , srv393159585:84 , srv418284352:84 , srv419539316:84 , srv421313952:84 , srv425833235:84 , srv42771850:84 , srv475265531:84 , srv526390088:84 , srv549487426:84 , srv588404093:84 , srv615958063:84 , srv644028:84 , srv647982473:84 , srv653549515:84 , srv690256121:84 , srv705276968:84 , srv734916047:84 , srv738699357:84 , srv768200392:84 , srv835699459:84 , srv835748444:84 , srv83965199:84 , srv840050903:84 , srv853702667:84 , srv904546334:84 , srv971506885:84 , srv982937166:84 , srv1149218026:85 , srv1425149658:85 , srv1866746757:85 , srv1819888605:684 } 2024-04-07 11:53:19,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:19,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1818631405=46, srv1864415740=54, srv1442075312=33, srv644028=84, srv647982473=85, srv1065115920=5, srv1425149658=31, srv835748444=93, srv853702667=96, srv1220030384=19, srv1428815295=32, srv425833235=76, srv1124883874=9, srv103426226=4, srv1329143850=27, srv1764968718=43, srv1466164083=35, srv835699459=92, srv1628561085=42, srv1321547570=26, srv1123587450=8, srv1016668458=1, srv1612018790=41, srv1188230381=13, srv1907863725=56, srv393159585=72, srv1919152937=57, srv904546334=97, srv1335289828=28, srv1785492637=44, srv1182941913=12, srv653549515=86, srv1817695757=45, srv1979798452=60, srv1834367081=49, srv1198526285=16, srv182564192=48, srv690256121=87, srv121954190=18, srv705276968=88, srv248511581=68, srv475265531=79, srv1258959704=20, srv1993973936=61, srv734916047=89, srv1850574473=50, srv2133709109=67, srv2083561613=64, srv1020825465=2, srv2032371974=62, srv83965199=94, srv1120346573=7, srv42771850=77, srv2107063290=66, srv1483884137=37, srv971506885=98, srv738699357=90, srv1819888605=47, srv1859157695=52, srv1551422092=39, srv1467237953=36, srv15787304=40, srv1193389815=15, srv295866794=69, srv362115625=70, srv840050903=95, srv1317200194=25, srv1212940832=17, srv1449835519=34, srv1003522253=0, srv1135533112=10, srv103044595=3, srv1854897827=51, srv2097211903=65, srv1149218026=11, srv419539316=74, srv421313952=75, srv526390088=80, srv2041257999=63, srv1967537971=58, srv1544100146=38, srv380059345=71, srv1418359096=30, srv1866746757=55, srv1297931594=22, srv549487426=81, srv1311693799=24, srv768200392=91, srv1098742374=6, srv443533973=78, srv1862055737=53, srv1292086771=21, srv197921918=59, srv1302126054=23, srv615958063=83, srv588404093=82, srv982937166=99, srv1192994323=14, srv1361114204=29, srv418284352=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-07 11:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-07 11:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-07 11:53:19,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-07 11:53:19,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-07 11:53:19,147 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:19,147 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1016668458:83 , srv1418359096:83 , srv443533973:83 , srv1003522253:84 , srv1020825465:84 , srv103044595:84 , srv103426226:84 , srv1065115920:84 , srv1098742374:84 , srv1120346573:84 , srv1123587450:84 , srv1124883874:84 , srv1135533112:84 , srv1182941913:84 , srv1188230381:84 , srv1192994323:84 , srv1193389815:84 , srv1198526285:84 , srv1212940832:84 , srv121954190:84 , srv1220030384:84 , srv1258959704:84 , srv1292086771:84 , srv1297931594:84 , srv1302126054:84 , srv1311693799:84 , srv1317200194:84 , srv1321547570:84 , srv1329143850:84 , srv1335289828:84 , srv1361114204:84 , srv1428815295:84 , srv1442075312:84 , srv1449835519:84 , srv1466164083:84 , srv1467237953:84 , srv1483884137:84 , srv1544100146:84 , srv1551422092:84 , srv15787304:84 , srv1612018790:84 , srv1628561085:84 , srv1764968718:84 , srv1785492637:84 , srv1817695757:84 , srv1818631405:84 , srv182564192:84 , srv1834367081:84 , srv1850574473:84 , srv1854897827:84 , srv1859157695:84 , srv1862055737:84 , srv1864415740:84 , srv1907863725:84 , srv1919152937:84 , srv1967537971:84 , srv197921918:84 , srv1979798452:84 , srv1993973936:84 , srv2032371974:84 , srv2041257999:84 , srv2083561613:84 , srv2097211903:84 , srv2107063290:84 , srv2133709109:84 , srv248511581:84 , srv295866794:84 , srv362115625:84 , srv380059345:84 , srv393159585:84 , srv418284352:84 , srv419539316:84 , srv421313952:84 , srv425833235:84 , srv42771850:84 , srv475265531:84 , srv526390088:84 , srv549487426:84 , srv588404093:84 , srv615958063:84 , srv644028:84 , srv647982473:84 , srv653549515:84 , srv690256121:84 , srv705276968:84 , srv734916047:84 , srv738699357:84 , srv768200392:84 , srv835699459:84 , srv835748444:84 , srv83965199:84 , srv840050903:84 , srv853702667:84 , srv904546334:84 , srv971506885:84 , srv982937166:84 , srv1149218026:85 , srv1425149658:85 , srv1866746757:85 , srv1819888605:684 } 2024-04-07 11:53:19,180 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=335 (was 346), ProcessCount=171 (was 171), AvailableMemoryMB=6686 (was 7453) 2024-04-07 11:53:19,188 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=335, ProcessCount=171, AvailableMemoryMB=6686 2024-04-07 11:53:19,189 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-07 11:53:19,189 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-07 11:53:19,202 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003407638:84 , srv1007164838:84 , srv1184037683:84 , srv1228088683:84 , srv1307184464:84 , srv1314002062:84 , srv1344739940:84 , srv1430563500:84 , srv1463018966:84 , srv1476911877:84 , srv1529995990:84 , srv1556907494:84 , srv1610211394:84 , srv1687065779:84 , srv1692015856:84 , srv1724573713:84 , srv1776026546:84 , srv1914300487:84 , srv1931434985:84 , srv198932392:84 , srv2024458787:84 , srv2055909273:84 , srv257752474:84 , srv382114062:84 , srv422128299:84 , srv590188562:84 , srv590867461:84 , srv837274192:84 , srv94373782:84 , srv1308311008:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-07 11:53:19,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:53:19,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1931434985=19, srv1344739940=7, srv1556907494=12, srv1007164838=1, srv422128299=25, srv1307184464=4, srv1003407638=0, srv1476911877=10, srv1184037683=2, srv590188562=26, srv590867461=27, srv1692015856=15, srv837274192=28, srv94373782=29, srv1308311008=5, srv1228088683=3, srv1314002062=6, srv1529995990=11, srv1430563500=8, srv1776026546=17, srv382114062=24, srv1914300487=18, srv198932392=20, srv1463018966=9, srv1724573713=16, srv257752474=23, srv1687065779=14, srv1610211394=13, srv2024458787=21, srv2055909273=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:53:19,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:53:19,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-07 11:53:19,225 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1264232800411504, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:53:19,225 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912591398891589, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1264232800411504, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-07 11:55:19,215 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5807870 different iterations. Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912591398891589 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:55:19,222 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1003407638:90 , srv1007164838:90 , srv1184037683:90 , srv1228088683:90 , srv1307184464:90 , srv1308311008:90 , srv1314002062:90 , srv1344739940:90 , srv1430563500:90 , srv1463018966:90 , srv1476911877:90 , srv1529995990:90 , srv1556907494:90 , srv1610211394:90 , srv1687065779:90 , srv1692015856:90 , srv1724573713:90 , srv1776026546:90 , srv1914300487:90 , srv1931434985:90 , srv198932392:90 , srv2024458787:90 , srv2055909273:90 , srv257752474:90 , srv382114062:90 , srv422128299:90 , srv590188562:90 , srv590867461:90 , srv837274192:90 , srv94373782:90 } 2024-04-07 11:55:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-07 11:55:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1931434985=19, srv1344739940=7, srv1556907494=12, srv1007164838=1, srv422128299=25, srv1307184464=4, srv1003407638=0, srv1476911877=10, srv1184037683=2, srv590188562=26, srv590867461=27, srv1692015856=15, srv837274192=28, srv94373782=29, srv1308311008=5, srv1228088683=3, srv1314002062=6, srv1529995990=11, srv1430563500=8, srv1776026546=17, srv382114062=24, srv1914300487=18, srv198932392=20, srv1463018966=9, srv1724573713=16, srv257752474=23, srv1687065779=14, srv1610211394=13, srv2024458787=21, srv2055909273=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-07 11:55:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-07 11:55:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-07 11:55:19,245 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-07 11:55:19,245 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1003407638:90 , srv1007164838:90 , srv1184037683:90 , srv1228088683:90 , srv1307184464:90 , srv1308311008:90 , srv1314002062:90 , srv1344739940:90 , srv1430563500:90 , srv1463018966:90 , srv1476911877:90 , srv1529995990:90 , srv1556907494:90 , srv1610211394:90 , srv1687065779:90 , srv1692015856:90 , srv1724573713:90 , srv1776026546:90 , srv1914300487:90 , srv1931434985:90 , srv198932392:90 , srv2024458787:90 , srv2055909273:90 , srv257752474:90 , srv382114062:90 , srv422128299:90 , srv590188562:90 , srv590867461:90 , srv837274192:90 , srv94373782:90 } 2024-04-07 11:55:19,261 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=288 (was 335), ProcessCount=167 (was 171), AvailableMemoryMB=5912 (was 6686)