2023-04-05 18:56:33,957 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-05 18:56:34,271 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:56:34,385 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-05 18:56:34,464 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-05 18:56:34,470 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-05 18:56:34,483 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=358, ProcessCount=169, AvailableMemoryMB=2390 2023-04-05 18:56:34,485 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-05 18:56:34,486 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-05 18:56:34,617 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100261077:84 , srv1012190103:84 , srv1015811103:84 , srv10187118:84 , srv1034678462:84 , srv1060890601:84 , srv1075684273:84 , srv1097338760:84 , srv1099471852:84 , srv1116328371:84 , srv1126136617:84 , srv114575389:84 , srv115382467:84 , srv1167160420:84 , srv118359240:84 , srv1202373097:84 , srv1208365735:84 , srv1211572341:84 , srv1232581366:84 , srv1256414150:84 , srv127344426:84 , srv1291931418:84 , srv1326935904:84 , srv1336275500:84 , srv1358587631:84 , srv1359098234:84 , srv1361554358:84 , srv1377339007:84 , srv1509320560:84 , srv1516273875:84 , srv1519723975:84 , srv1523830849:84 , srv1526752701:84 , srv1542549212:84 , srv158696051:84 , srv159587296:84 , srv161016961:84 , srv1630013629:84 , srv1666935102:84 , srv1674390751:84 , srv1695566171:84 , srv1727072189:84 , srv1744538914:84 , srv1783822231:84 , srv1793182716:84 , srv1793423888:84 , srv1800437159:84 , srv1888901576:84 , srv1913874522:84 , srv1925968808:84 , srv1945650558:84 , srv1957581594:84 , srv1975970269:84 , srv2048770748:84 , srv2058744914:84 , srv2089365322:84 , srv2091400595:84 , srv2093127508:84 , srv2116791356:84 , srv2127853286:84 , srv2133512018:84 , srv2136754306:84 , srv2137313922:84 , srv2140828105:84 , srv2145864155:84 , srv2146680835:84 , srv276274186:84 , srv356672241:84 , srv383655127:84 , srv384630008:84 , srv408782120:84 , srv423981446:84 , srv436264223:84 , srv438868882:84 , srv476770720:84 , srv491354523:84 , srv491490336:84 , srv531836813:84 , srv566423057:84 , srv629033860:84 , srv637480892:84 , srv660221218:84 , srv67241908:84 , srv710542930:84 , srv723706080:84 , srv76829078:84 , srv769346945:84 , srv778458209:84 , srv788730253:84 , srv826029719:84 , srv833786325:84 , srv859393448:84 , srv867474957:84 , srv86938415:84 , srv918439770:84 , srv925694244:84 , srv954979401:84 , srv964584404:84 , srv971644127:84 , srv128602445:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-05 18:56:34,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:34,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv159587296=36, srv637480892=81, srv2048770748=54, srv1519723975=31, srv161016961=37, srv128602445=21, srv1523830849=32, srv1211572341=17, srv1336275500=24, srv2093127508=58, srv1167160420=13, srv1783822231=44, srv1060890601=5, srv2136754306=62, srv1793423888=46, srv925694244=96, srv1793182716=45, srv115382467=12, srv1800437159=47, srv1377339007=28, srv1291931418=22, srv1126136617=10, srv1975970269=53, srv1516273875=30, srv408782120=71, srv127344426=20, srv867474957=93, srv859393448=92, srv1913874522=49, srv67241908=83, srv86938415=94, srv566423057=79, srv1202373097=15, srv1695566171=41, srv158696051=35, srv660221218=82, srv384630008=70, srv356672241=68, srv476770720=75, srv491490336=77, srv629033860=80, srv1015811103=2, srv1727072189=42, srv778458209=88, srv1359098234=26, srv954979401=97, srv1526752701=33, srv1097338760=7, srv276274186=67, srv1674390751=40, srv423981446=72, srv1256414150=19, srv788730253=89, srv1012190103=1, srv2140828105=64, srv723706080=85, srv2133512018=61, srv2116791356=59, srv1326935904=23, srv833786325=91, srv964584404=98, srv383655127=69, srv76829078=86, srv1744538914=43, srv118359240=14, srv1116328371=9, srv918439770=95, srv2137313922=63, srv1925968808=50, srv114575389=11, srv1208365735=16, srv1957581594=52, srv491354523=76, srv1361554358=27, srv1232581366=18, srv100261077=0, srv2127853286=60, srv2058744914=55, srv2145864155=65, srv1542549212=34, srv1666935102=39, srv2091400595=57, srv531836813=78, srv2146680835=66, srv2089365322=56, srv1034678462=4, srv710542930=84, srv826029719=90, srv1358587631=25, srv10187118=3, srv1075684273=6, srv971644127=99, srv436264223=73, srv1509320560=29, srv1099471852=8, srv769346945=87, srv438868882=74, srv1945650558=51, srv1630013629=38, srv1888901576=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:34,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:34,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:34,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-05 18:56:34,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-05 18:56:34,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-05 18:56:34,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-05 18:56:34,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-05 18:56:34,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-05 18:56:34,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-05 18:56:34,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-05 18:56:34,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-05 18:56:34,880 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:34,880 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-05 18:56:34,881 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-05 18:56:35,179 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations. Found a solution that moves 4691 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17872234068317167. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04061290012810048, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05646329121446891, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5212222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04061290012810048, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17433333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2365, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:35,201 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv660221218:81 , srv769346945:82 , srv1075684273:83 , srv1516273875:83 , srv159587296:83 , srv1666935102:83 , srv1800437159:83 , srv1945650558:83 , srv2093127508:83 , srv383655127:83 , srv566423057:83 , srv76829078:83 , srv1015811103:84 , srv1126136617:84 , srv118359240:84 , srv1358587631:84 , srv1793423888:84 , srv1913874522:84 , srv2048770748:84 , srv2058744914:84 , srv2127853286:84 , srv2136754306:84 , srv476770720:84 , srv867474957:84 , srv1034678462:85 , srv114575389:85 , srv127344426:85 , srv158696051:85 , srv2116791356:85 , srv2140828105:85 , srv356672241:85 , srv384630008:85 , srv723706080:85 , srv971644127:85 , srv1060890601:86 , srv1116328371:86 , srv1208365735:86 , srv1291931418:86 , srv1336275500:86 , srv1359098234:86 , srv1361554358:86 , srv1523830849:86 , srv1542549212:86 , srv161016961:86 , srv1630013629:86 , srv1783822231:86 , srv1957581594:86 , srv2089365322:86 , srv2133512018:86 , srv2146680835:86 , srv276274186:86 , srv491354523:86 , srv491490336:86 , srv637480892:86 , srv778458209:86 , srv859393448:86 , srv954979401:86 , srv1012190103:87 , srv1099471852:87 , srv115382467:87 , srv1202373097:87 , srv1211572341:87 , srv1377339007:87 , srv1509320560:87 , srv1674390751:87 , srv1727072189:87 , srv1793182716:87 , srv1925968808:87 , srv2091400595:87 , srv2137313922:87 , srv408782120:87 , srv423981446:87 , srv438868882:87 , srv67241908:87 , srv710542930:87 , srv918439770:87 , srv925694244:87 , srv964584404:87 , srv1256414150:88 , srv1326935904:88 , srv1975970269:88 , srv436264223:88 , srv531836813:88 , srv1097338760:89 , srv1167160420:89 , srv629033860:89 , srv1232581366:90 , srv1695566171:90 , srv1744538914:90 , srv1888901576:90 , srv826029719:90 , srv833786325:90 , srv100261077:91 , srv1519723975:91 , srv1526752701:92 , srv2145864155:92 , srv86938415:92 , srv10187118:93 , srv788730253:95 , srv128602445:451 } 2023-04-05 18:56:35,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:35,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv159587296=36, srv637480892=81, srv2048770748=54, srv1519723975=31, srv161016961=37, srv128602445=21, srv1523830849=32, srv1211572341=17, srv1336275500=24, srv2093127508=58, srv1167160420=13, srv1783822231=44, srv1060890601=5, srv2136754306=62, srv1793423888=46, srv925694244=96, srv1793182716=45, srv115382467=12, srv1800437159=47, srv1377339007=28, srv1291931418=22, srv1126136617=10, srv1975970269=53, srv1516273875=30, srv408782120=71, srv127344426=20, srv867474957=93, srv859393448=92, srv1913874522=49, srv67241908=83, srv86938415=94, srv566423057=79, srv1202373097=15, srv1695566171=41, srv158696051=35, srv660221218=82, srv384630008=70, srv356672241=68, srv476770720=75, srv491490336=77, srv629033860=80, srv1015811103=2, srv1727072189=42, srv778458209=88, srv1359098234=26, srv954979401=97, srv1526752701=33, srv1097338760=7, srv276274186=67, srv1674390751=40, srv423981446=72, srv1256414150=19, srv788730253=89, srv1012190103=1, srv2140828105=64, srv723706080=85, srv2133512018=61, srv2116791356=59, srv1326935904=23, srv833786325=91, srv964584404=98, srv383655127=69, srv76829078=86, srv1744538914=43, srv118359240=14, srv1116328371=9, srv918439770=95, srv2137313922=63, srv1925968808=50, srv114575389=11, srv1208365735=16, srv1957581594=52, srv491354523=76, srv1361554358=27, srv1232581366=18, srv100261077=0, srv2127853286=60, srv2058744914=55, srv2145864155=65, srv1542549212=34, srv1666935102=39, srv2091400595=57, srv531836813=78, srv2146680835=66, srv2089365322=56, srv1034678462=4, srv710542930=84, srv826029719=90, srv1358587631=25, srv10187118=3, srv1075684273=6, srv971644127=99, srv436264223=73, srv1509320560=29, srv1099471852=8, srv769346945=87, srv438868882=74, srv1945650558=51, srv1630013629=38, srv1888901576=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:35,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:35,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-05 18:56:35,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-05 18:56:35,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-05 18:56:35,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-05 18:56:35,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-05 18:56:35,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-05 18:56:35,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-05 18:56:35,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-05 18:56:35,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-05 18:56:35,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-05 18:56:35,374 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04061290012810048, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05646329121446891, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04061290012810048, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17433333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2365, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:35,374 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-05 18:56:35,374 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17868949359911687, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04061290012810048, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05646329121446891, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04061290012810048, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17433333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2365, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-05 18:56:35,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 450 ms to try 5000 different iterations. Found a solution that moves 2594 regions; Going from a computed imbalance of 0.17868949359911687 to a new imbalance of 0.017122638224438453. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02304590213438561); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031835012383663576, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2882222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02304590213438561); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:35,706 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2145864155:78 , srv788730253:78 , srv100261077:79 , srv10187118:79 , srv1695566171:80 , srv1232581366:81 , srv1519723975:81 , srv1526752701:81 , srv86938415:81 , srv1097338760:82 , srv1888901576:82 , srv629033860:82 , srv1211572341:83 , srv1361554358:83 , srv1509320560:83 , srv1744538914:83 , srv436264223:83 , srv438868882:83 , srv531836813:83 , srv826029719:83 , srv833786325:83 , srv1202373097:84 , srv1326935904:84 , srv859393448:84 , srv1012190103:85 , srv1015811103:85 , srv1099471852:85 , srv1167160420:85 , srv1208365735:85 , srv1256414150:85 , srv1291931418:85 , srv1359098234:85 , srv1727072189:85 , srv1925968808:85 , srv1975970269:85 , srv2091400595:85 , srv2137313922:85 , srv2146680835:85 , srv276274186:85 , srv408782120:85 , srv423981446:85 , srv918439770:85 , srv964584404:85 , srv1034678462:86 , srv115382467:86 , srv1358587631:86 , srv1542549212:86 , srv161016961:86 , srv1630013629:86 , srv1783822231:86 , srv2133512018:86 , srv384630008:86 , srv491490336:86 , srv637480892:86 , srv67241908:86 , srv710542930:86 , srv723706080:86 , srv778458209:86 , srv954979401:86 , srv1060890601:87 , srv1126136617:87 , srv127344426:87 , srv1336275500:87 , srv1377339007:87 , srv1516273875:87 , srv1523830849:87 , srv1674390751:87 , srv1793182716:87 , srv1957581594:87 , srv2089365322:87 , srv2093127508:87 , srv2116791356:87 , srv2136754306:87 , srv356672241:87 , srv491354523:87 , srv76829078:87 , srv925694244:87 , srv1075684273:88 , srv1116328371:88 , srv118359240:88 , srv1666935102:88 , srv1793423888:88 , srv1913874522:88 , srv2048770748:88 , srv2058744914:88 , srv2127853286:88 , srv383655127:88 , srv971644127:88 , srv114575389:89 , srv158696051:89 , srv159587296:89 , srv1945650558:89 , srv2140828105:89 , srv476770720:89 , srv566423057:89 , srv1800437159:90 , srv769346945:90 , srv867474957:90 , srv660221218:91 , srv128602445:528 } 2023-04-05 18:56:35,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:35,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv159587296=36, srv637480892=81, srv2048770748=54, srv1519723975=31, srv161016961=37, srv128602445=21, srv1523830849=32, srv1211572341=17, srv1336275500=24, srv2093127508=58, srv1167160420=13, srv1783822231=44, srv1060890601=5, srv2136754306=62, srv1793423888=46, srv925694244=96, srv1793182716=45, srv115382467=12, srv1800437159=47, srv1377339007=28, srv1291931418=22, srv1126136617=10, srv1975970269=53, srv1516273875=30, srv408782120=71, srv127344426=20, srv867474957=93, srv859393448=92, srv1913874522=49, srv67241908=83, srv86938415=94, srv566423057=79, srv1202373097=15, srv1695566171=41, srv158696051=35, srv660221218=82, srv384630008=70, srv356672241=68, srv476770720=75, srv491490336=77, srv629033860=80, srv1015811103=2, srv1727072189=42, srv778458209=88, srv1359098234=26, srv954979401=97, srv1526752701=33, srv1097338760=7, srv276274186=67, srv1674390751=40, srv423981446=72, srv1256414150=19, srv788730253=89, srv1012190103=1, srv2140828105=64, srv723706080=85, srv2133512018=61, srv2116791356=59, srv1326935904=23, srv833786325=91, srv964584404=98, srv383655127=69, srv76829078=86, srv1744538914=43, srv118359240=14, srv1116328371=9, srv918439770=95, srv2137313922=63, srv1925968808=50, srv114575389=11, srv1208365735=16, srv1957581594=52, srv491354523=76, srv1361554358=27, srv1232581366=18, srv100261077=0, srv2127853286=60, srv2058744914=55, srv2145864155=65, srv1542549212=34, srv1666935102=39, srv2091400595=57, srv531836813=78, srv2146680835=66, srv2089365322=56, srv1034678462=4, srv710542930=84, srv826029719=90, srv1358587631=25, srv10187118=3, srv1075684273=6, srv971644127=99, srv436264223=73, srv1509320560=29, srv1099471852=8, srv769346945=87, srv438868882=74, srv1945650558=51, srv1630013629=38, srv1888901576=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:35,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:35,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:35,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:35,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:35,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-05 18:56:35,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-05 18:56:35,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-05 18:56:35,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-05 18:56:35,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-05 18:56:35,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-05 18:56:35,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-05 18:56:35,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-05 18:56:35,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-05 18:56:35,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-05 18:56:35,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-05 18:56:35,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-05 18:56:35,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-05 18:56:35,901 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02304590213438561); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031835012383663576, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02304590213438561); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:35,901 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-05 18:56:35,901 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017104474648220555, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02304590213438561); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031835012383663576, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02304590213438561); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-05 18:56:36,187 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 418 ms to try 5000 different iterations. Found a solution that moves 963 regions; Going from a computed imbalance of 0.017104474648220555 to a new imbalance of 4.8674119656410544E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0056567660698823964); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007247430753394787); MoveCostFunction : (multiplier=7.0, imbalance=0.107, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0056567660698823964); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:36,189 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1167160420:83 , srv1377339007:83 , srv2089365322:83 , srv867474957:83 , srv1060890601:84 , srv114575389:84 , srv1326935904:84 , srv1336275500:84 , srv1523830849:84 , srv1526752701:84 , srv158696051:84 , srv1674390751:84 , srv1793182716:84 , srv1957581594:84 , srv1975970269:84 , srv2116791356:84 , srv2140828105:84 , srv356672241:84 , srv476770720:84 , srv67241908:84 , srv788730253:84 , srv925694244:84 , srv1012190103:85 , srv10187118:85 , srv1116328371:85 , srv115382467:85 , srv1256414150:85 , srv1542549212:85 , srv159587296:85 , srv1727072189:85 , srv1744538914:85 , srv1783822231:85 , srv1793423888:85 , srv1800437159:85 , srv1945650558:85 , srv2048770748:85 , srv2091400595:85 , srv2127853286:85 , srv491490336:85 , srv637480892:85 , srv710542930:85 , srv769346945:85 , srv778458209:85 , srv826029719:85 , srv833786325:85 , srv86938415:85 , srv918439770:85 , srv954979401:85 , srv964584404:85 , srv971644127:85 , srv1034678462:86 , srv1097338760:86 , srv1099471852:86 , srv1126136617:86 , srv118359240:86 , srv1202373097:86 , srv1208365735:86 , srv1232581366:86 , srv127344426:86 , srv1519723975:86 , srv161016961:86 , srv1630013629:86 , srv1888901576:86 , srv1913874522:86 , srv1925968808:86 , srv2058744914:86 , srv2133512018:86 , srv2137313922:86 , srv276274186:86 , srv383655127:86 , srv408782120:86 , srv423981446:86 , srv566423057:86 , srv629033860:86 , srv660221218:86 , srv723706080:86 , srv1075684273:87 , srv1211572341:87 , srv1291931418:87 , srv1359098234:87 , srv1509320560:87 , srv1666935102:87 , srv2136754306:87 , srv2145864155:87 , srv2146680835:87 , srv384630008:87 , srv436264223:87 , srv438868882:87 , srv491354523:87 , srv531836813:87 , srv859393448:87 , srv100261077:88 , srv1358587631:88 , srv1516273875:88 , srv1695566171:88 , srv2093127508:88 , srv76829078:88 , srv1015811103:89 , srv1361554358:89 , srv128602445:529 } 2023-04-05 18:56:36,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:36,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv159587296=36, srv637480892=81, srv2048770748=54, srv1519723975=31, srv161016961=37, srv128602445=21, srv1523830849=32, srv1211572341=17, srv1336275500=24, srv2093127508=58, srv1167160420=13, srv1783822231=44, srv1060890601=5, srv2136754306=62, srv1793423888=46, srv925694244=96, srv1793182716=45, srv115382467=12, srv1800437159=47, srv1377339007=28, srv1291931418=22, srv1126136617=10, srv1975970269=53, srv1516273875=30, srv408782120=71, srv127344426=20, srv867474957=93, srv859393448=92, srv1913874522=49, srv67241908=83, srv86938415=94, srv566423057=79, srv1202373097=15, srv1695566171=41, srv158696051=35, srv660221218=82, srv384630008=70, srv356672241=68, srv476770720=75, srv491490336=77, srv629033860=80, srv1015811103=2, srv1727072189=42, srv778458209=88, srv1359098234=26, srv954979401=97, srv1526752701=33, srv1097338760=7, srv276274186=67, srv1674390751=40, srv423981446=72, srv1256414150=19, srv788730253=89, srv1012190103=1, srv2140828105=64, srv723706080=85, srv2133512018=61, srv2116791356=59, srv1326935904=23, srv833786325=91, srv964584404=98, srv383655127=69, srv76829078=86, srv1744538914=43, srv118359240=14, srv1116328371=9, srv918439770=95, srv2137313922=63, srv1925968808=50, srv114575389=11, srv1208365735=16, srv1957581594=52, srv491354523=76, srv1361554358=27, srv1232581366=18, srv100261077=0, srv2127853286=60, srv2058744914=55, srv2145864155=65, srv1542549212=34, srv1666935102=39, srv2091400595=57, srv531836813=78, srv2146680835=66, srv2089365322=56, srv1034678462=4, srv710542930=84, srv826029719=90, srv1358587631=25, srv10187118=3, srv1075684273=6, srv971644127=99, srv436264223=73, srv1509320560=29, srv1099471852=8, srv769346945=87, srv438868882=74, srv1945650558=51, srv1630013629=38, srv1888901576=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:36,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:36,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:36,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:36,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-05 18:56:36,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-05 18:56:36,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-05 18:56:36,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-05 18:56:36,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-05 18:56:36,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-05 18:56:36,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-05 18:56:36,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-05 18:56:36,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-05 18:56:36,339 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0056567660698823964); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007247430753394787); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0056567660698823964); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:36,340 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-05 18:56:36,340 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.799981264415778E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0056567660698823964); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007247430753394787); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0056567660698823964); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-05 18:56:36,703 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 468 ms to try 5000 different iterations. Found a solution that moves 194 regions; Going from a computed imbalance of 4.799981264415778E-4 to a new imbalance of 2.6748098194439246E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.021555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:36,704 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116328371:83 , srv1358587631:83 , srv161016961:83 , srv384630008:83 , srv491354523:83 , srv100261077:84 , srv1015811103:84 , srv1075684273:84 , srv1099471852:84 , srv1126136617:84 , srv114575389:84 , srv115382467:84 , srv1167160420:84 , srv118359240:84 , srv1202373097:84 , srv1256414150:84 , srv127344426:84 , srv1291931418:84 , srv1359098234:84 , srv1361554358:84 , srv1519723975:84 , srv158696051:84 , srv1630013629:84 , srv1666935102:84 , srv1674390751:84 , srv1695566171:84 , srv1744538914:84 , srv1793182716:84 , srv1800437159:84 , srv1888901576:84 , srv1913874522:84 , srv1925968808:84 , srv2091400595:84 , srv2093127508:84 , srv2133512018:84 , srv2136754306:84 , srv2137313922:84 , srv2140828105:84 , srv2146680835:84 , srv408782120:84 , srv423981446:84 , srv436264223:84 , srv491490336:84 , srv531836813:84 , srv566423057:84 , srv660221218:84 , srv710542930:84 , srv76829078:84 , srv788730253:84 , srv833786325:84 , srv86938415:84 , srv925694244:84 , srv971644127:84 , srv1012190103:85 , srv10187118:85 , srv1060890601:85 , srv1097338760:85 , srv1208365735:85 , srv1211572341:85 , srv1232581366:85 , srv1336275500:85 , srv1377339007:85 , srv1509320560:85 , srv1516273875:85 , srv1526752701:85 , srv1542549212:85 , srv159587296:85 , srv1783822231:85 , srv1793423888:85 , srv1975970269:85 , srv2048770748:85 , srv2058744914:85 , srv2089365322:85 , srv2127853286:85 , srv2145864155:85 , srv276274186:85 , srv383655127:85 , srv438868882:85 , srv476770720:85 , srv629033860:85 , srv637480892:85 , srv67241908:85 , srv723706080:85 , srv769346945:85 , srv778458209:85 , srv826029719:85 , srv859393448:85 , srv867474957:85 , srv918439770:85 , srv954979401:85 , srv964584404:85 , srv1034678462:86 , srv1326935904:86 , srv1523830849:86 , srv1727072189:86 , srv1945650558:86 , srv1957581594:86 , srv2116791356:86 , srv356672241:86 , srv128602445:635 } 2023-04-05 18:56:36,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:36,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv159587296=36, srv637480892=81, srv2048770748=54, srv1519723975=31, srv161016961=37, srv128602445=21, srv1523830849=32, srv1211572341=17, srv1336275500=24, srv2093127508=58, srv1167160420=13, srv1783822231=44, srv1060890601=5, srv2136754306=62, srv1793423888=46, srv925694244=96, srv1793182716=45, srv115382467=12, srv1800437159=47, srv1377339007=28, srv1291931418=22, srv1126136617=10, srv1975970269=53, srv1516273875=30, srv408782120=71, srv127344426=20, srv867474957=93, srv859393448=92, srv1913874522=49, srv67241908=83, srv86938415=94, srv566423057=79, srv1202373097=15, srv1695566171=41, srv158696051=35, srv660221218=82, srv384630008=70, srv356672241=68, srv476770720=75, srv491490336=77, srv629033860=80, srv1015811103=2, srv1727072189=42, srv778458209=88, srv1359098234=26, srv954979401=97, srv1526752701=33, srv1097338760=7, srv276274186=67, srv1674390751=40, srv423981446=72, srv1256414150=19, srv788730253=89, srv1012190103=1, srv2140828105=64, srv723706080=85, srv2133512018=61, srv2116791356=59, srv1326935904=23, srv833786325=91, srv964584404=98, srv383655127=69, srv76829078=86, srv1744538914=43, srv118359240=14, srv1116328371=9, srv918439770=95, srv2137313922=63, srv1925968808=50, srv114575389=11, srv1208365735=16, srv1957581594=52, srv491354523=76, srv1361554358=27, srv1232581366=18, srv100261077=0, srv2127853286=60, srv2058744914=55, srv2145864155=65, srv1542549212=34, srv1666935102=39, srv2091400595=57, srv531836813=78, srv2146680835=66, srv2089365322=56, srv1034678462=4, srv710542930=84, srv826029719=90, srv1358587631=25, srv10187118=3, srv1075684273=6, srv971644127=99, srv436264223=73, srv1509320560=29, srv1099471852=8, srv769346945=87, srv438868882=74, srv1945650558=51, srv1630013629=38, srv1888901576=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:36,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:36,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:36,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-05 18:56:36,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-05 18:56:36,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-05 18:56:36,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-05 18:56:36,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-05 18:56:36,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-05 18:56:36,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-05 18:56:36,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-05 18:56:36,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-05 18:56:36,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-05 18:56:36,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-05 18:56:36,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-05 18:56:36,849 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:36,850 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-05 18:56:36,850 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.53896811604098E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-05 18:56:37,075 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 325 ms to try 5000 different iterations. Found a solution that moves 20 regions; Going from a computed imbalance of 2.53896811604098E-5 to a new imbalance of 7.527800493511471E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0022222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:37,076 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1034678462:83 , srv1516273875:83 , srv1523830849:83 , srv1727072189:83 , srv1945650558:83 , srv1957581594:83 , srv2058744914:83 , srv491354523:83 , srv826029719:83 , srv100261077:84 , srv1012190103:84 , srv1015811103:84 , srv10187118:84 , srv1060890601:84 , srv1075684273:84 , srv1097338760:84 , srv1099471852:84 , srv1116328371:84 , srv114575389:84 , srv115382467:84 , srv118359240:84 , srv1208365735:84 , srv1211572341:84 , srv1232581366:84 , srv1256414150:84 , srv127344426:84 , srv1291931418:84 , srv1326935904:84 , srv1336275500:84 , srv1359098234:84 , srv1361554358:84 , srv1377339007:84 , srv1509320560:84 , srv1519723975:84 , srv1526752701:84 , srv1542549212:84 , srv158696051:84 , srv159587296:84 , srv1630013629:84 , srv1666935102:84 , srv1674390751:84 , srv1695566171:84 , srv1744538914:84 , srv1783822231:84 , srv1793182716:84 , srv1793423888:84 , srv1800437159:84 , srv1888901576:84 , srv1913874522:84 , srv1925968808:84 , srv1975970269:84 , srv2048770748:84 , srv2093127508:84 , srv2116791356:84 , srv2127853286:84 , srv2133512018:84 , srv2136754306:84 , srv2137313922:84 , srv2140828105:84 , srv2145864155:84 , srv2146680835:84 , srv276274186:84 , srv356672241:84 , srv383655127:84 , srv408782120:84 , srv423981446:84 , srv436264223:84 , srv438868882:84 , srv476770720:84 , srv531836813:84 , srv566423057:84 , srv629033860:84 , srv637480892:84 , srv660221218:84 , srv67241908:84 , srv710542930:84 , srv723706080:84 , srv76829078:84 , srv769346945:84 , srv778458209:84 , srv833786325:84 , srv859393448:84 , srv867474957:84 , srv86938415:84 , srv918439770:84 , srv925694244:84 , srv954979401:84 , srv964584404:84 , srv971644127:84 , srv1126136617:85 , srv1167160420:85 , srv1202373097:85 , srv1358587631:85 , srv161016961:85 , srv2089365322:85 , srv2091400595:85 , srv384630008:85 , srv491490336:85 , srv788730253:85 , srv128602445:683 } 2023-04-05 18:56:37,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:37,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv159587296=36, srv637480892=81, srv2048770748=54, srv1519723975=31, srv161016961=37, srv128602445=21, srv1523830849=32, srv1211572341=17, srv1336275500=24, srv2093127508=58, srv1167160420=13, srv1783822231=44, srv1060890601=5, srv2136754306=62, srv1793423888=46, srv925694244=96, srv1793182716=45, srv115382467=12, srv1800437159=47, srv1377339007=28, srv1291931418=22, srv1126136617=10, srv1975970269=53, srv1516273875=30, srv408782120=71, srv127344426=20, srv867474957=93, srv859393448=92, srv1913874522=49, srv67241908=83, srv86938415=94, srv566423057=79, srv1202373097=15, srv1695566171=41, srv158696051=35, srv660221218=82, srv384630008=70, srv356672241=68, srv476770720=75, srv491490336=77, srv629033860=80, srv1015811103=2, srv1727072189=42, srv778458209=88, srv1359098234=26, srv954979401=97, srv1526752701=33, srv1097338760=7, srv276274186=67, srv1674390751=40, srv423981446=72, srv1256414150=19, srv788730253=89, srv1012190103=1, srv2140828105=64, srv723706080=85, srv2133512018=61, srv2116791356=59, srv1326935904=23, srv833786325=91, srv964584404=98, srv383655127=69, srv76829078=86, srv1744538914=43, srv118359240=14, srv1116328371=9, srv918439770=95, srv2137313922=63, srv1925968808=50, srv114575389=11, srv1208365735=16, srv1957581594=52, srv491354523=76, srv1361554358=27, srv1232581366=18, srv100261077=0, srv2127853286=60, srv2058744914=55, srv2145864155=65, srv1542549212=34, srv1666935102=39, srv2091400595=57, srv531836813=78, srv2146680835=66, srv2089365322=56, srv1034678462=4, srv710542930=84, srv826029719=90, srv1358587631=25, srv10187118=3, srv1075684273=6, srv971644127=99, srv436264223=73, srv1509320560=29, srv1099471852=8, srv769346945=87, srv438868882=74, srv1945650558=51, srv1630013629=38, srv1888901576=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:37,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:37,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:37,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:37,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:37,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:37,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:37,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-05 18:56:37,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-05 18:56:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-05 18:56:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-05 18:56:37,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-05 18:56:37,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-05 18:56:37,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-05 18:56:37,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-05 18:56:37,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-05 18:56:37,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-05 18:56:37,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-05 18:56:37,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-05 18:56:37,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-05 18:56:37,232 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:37,232 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1034678462:83 , srv1516273875:83 , srv1523830849:83 , srv1727072189:83 , srv1945650558:83 , srv1957581594:83 , srv2058744914:83 , srv491354523:83 , srv826029719:83 , srv100261077:84 , srv1012190103:84 , srv1015811103:84 , srv10187118:84 , srv1060890601:84 , srv1075684273:84 , srv1097338760:84 , srv1099471852:84 , srv1116328371:84 , srv114575389:84 , srv115382467:84 , srv118359240:84 , srv1208365735:84 , srv1211572341:84 , srv1232581366:84 , srv1256414150:84 , srv127344426:84 , srv1291931418:84 , srv1326935904:84 , srv1336275500:84 , srv1359098234:84 , srv1361554358:84 , srv1377339007:84 , srv1509320560:84 , srv1519723975:84 , srv1526752701:84 , srv1542549212:84 , srv158696051:84 , srv159587296:84 , srv1630013629:84 , srv1666935102:84 , srv1674390751:84 , srv1695566171:84 , srv1744538914:84 , srv1783822231:84 , srv1793182716:84 , srv1793423888:84 , srv1800437159:84 , srv1888901576:84 , srv1913874522:84 , srv1925968808:84 , srv1975970269:84 , srv2048770748:84 , srv2093127508:84 , srv2116791356:84 , srv2127853286:84 , srv2133512018:84 , srv2136754306:84 , srv2137313922:84 , srv2140828105:84 , srv2145864155:84 , srv2146680835:84 , srv276274186:84 , srv356672241:84 , srv383655127:84 , srv408782120:84 , srv423981446:84 , srv436264223:84 , srv438868882:84 , srv476770720:84 , srv531836813:84 , srv566423057:84 , srv629033860:84 , srv637480892:84 , srv660221218:84 , srv67241908:84 , srv710542930:84 , srv723706080:84 , srv76829078:84 , srv769346945:84 , srv778458209:84 , srv833786325:84 , srv859393448:84 , srv867474957:84 , srv86938415:84 , srv918439770:84 , srv925694244:84 , srv954979401:84 , srv964584404:84 , srv971644127:84 , srv1126136617:85 , srv1167160420:85 , srv1202373097:85 , srv1358587631:85 , srv161016961:85 , srv2089365322:85 , srv2091400595:85 , srv384630008:85 , srv491490336:85 , srv788730253:85 , srv128602445:683 } 2023-04-05 18:56:37,255 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=369 (was 358) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1848 (was 2390) 2023-04-05 18:56:37,262 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=369, ProcessCount=169, AvailableMemoryMB=1847 2023-04-05 18:56:37,262 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-05 18:56:37,263 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-05 18:56:37,270 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1028080898:84 , srv1125632272:84 , srv1151870725:84 , srv1167301405:84 , srv1178572496:84 , srv1208972162:84 , srv1233976197:84 , srv1360739475:84 , srv1505066015:84 , srv1519448080:84 , srv1621975330:84 , srv1665251800:84 , srv1689968291:84 , srv1795780983:84 , srv1913572648:84 , srv1997555200:84 , srv2008877423:84 , srv2027155092:84 , srv2071802069:84 , srv216724753:84 , srv267048729:84 , srv410555841:84 , srv422456242:84 , srv534431923:84 , srv668242522:84 , srv864407396:84 , srv879086974:84 , srv895160035:84 , srv934437829:84 , srv1566067132:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-05 18:56:37,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:56:37,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv422456242=23, srv1997555200=16, srv1233976197=6, srv2071802069=19, srv216724753=20, srv1125632272=1, srv534431923=24, srv2008877423=17, srv267048729=21, srv1519448080=9, srv879086974=27, srv1151870725=2, srv1665251800=12, srv1167301405=3, srv1913572648=15, srv864407396=26, srv410555841=22, srv1621975330=11, srv1795780983=14, srv1360739475=7, srv934437829=29, srv1178572496=4, srv1689968291=13, srv2027155092=18, srv895160035=28, srv668242522=25, srv1028080898=0, srv1566067132=10, srv1505066015=8, srv1208972162=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:56:37,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:56:37,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-05 18:56:37,297 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.124039837747527, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:56:37,297 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912583888742749, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.124039837747527, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-05 18:58:37,283 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6212232 different iterations. Found a solution that moves 1857 regions; Going from a computed imbalance of 0.9912583888742749 to a new imbalance of 4.334330639506329E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:58:37,290 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1028080898:90 , srv1125632272:90 , srv1151870725:90 , srv1167301405:90 , srv1178572496:90 , srv1208972162:90 , srv1233976197:90 , srv1360739475:90 , srv1505066015:90 , srv1519448080:90 , srv1566067132:90 , srv1621975330:90 , srv1665251800:90 , srv1689968291:90 , srv1795780983:90 , srv1913572648:90 , srv1997555200:90 , srv2008877423:90 , srv2027155092:90 , srv2071802069:90 , srv216724753:90 , srv267048729:90 , srv410555841:90 , srv422456242:90 , srv534431923:90 , srv668242522:90 , srv864407396:90 , srv879086974:90 , srv895160035:90 , srv934437829:90 } 2023-04-05 18:58:37,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-05 18:58:37,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv422456242=23, srv1997555200=16, srv1233976197=6, srv2071802069=19, srv216724753=20, srv1125632272=1, srv534431923=24, srv2008877423=17, srv267048729=21, srv1519448080=9, srv879086974=27, srv1151870725=2, srv1665251800=12, srv1167301405=3, srv1913572648=15, srv864407396=26, srv410555841=22, srv1621975330=11, srv1795780983=14, srv1360739475=7, srv934437829=29, srv1178572496=4, srv1689968291=13, srv2027155092=18, srv895160035=28, srv668242522=25, srv1028080898=0, srv1566067132=10, srv1505066015=8, srv1208972162=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-05 18:58:37,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-05 18:58:37,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-05 18:58:37,309 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-05 18:58:37,309 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1028080898:90 , srv1125632272:90 , srv1151870725:90 , srv1167301405:90 , srv1178572496:90 , srv1208972162:90 , srv1233976197:90 , srv1360739475:90 , srv1505066015:90 , srv1519448080:90 , srv1566067132:90 , srv1621975330:90 , srv1665251800:90 , srv1689968291:90 , srv1795780983:90 , srv1913572648:90 , srv1997555200:90 , srv2008877423:90 , srv2027155092:90 , srv2071802069:90 , srv216724753:90 , srv267048729:90 , srv410555841:90 , srv422456242:90 , srv534431923:90 , srv668242522:90 , srv864407396:90 , srv879086974:90 , srv895160035:90 , srv934437829:90 } 2023-04-05 18:58:37,321 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=390 (was 369) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2477 (was 1847) - AvailableMemoryMB LEAK? -