2023-05-23 00:00:21,961 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-23 00:00:22,198 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-23 00:00:22,296 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-23 00:00:22,351 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-23 00:00:22,359 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-23 00:00:22,371 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=121, ProcessCount=170, AvailableMemoryMB=8156 2023-05-23 00:00:22,373 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-23 00:00:22,373 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-23 00:00:22,507 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1053615599:84 , srv1054761085:84 , srv1071314334:84 , srv1081110801:84 , srv1087365607:84 , srv1101862756:84 , srv1138436396:84 , srv1145868826:84 , srv1146911296:84 , srv1156922055:84 , srv1169127021:84 , srv1181206326:84 , srv1183751389:84 , srv1207140819:84 , srv1233330773:84 , srv1257689904:84 , srv129552235:84 , srv1347138386:84 , srv1360033009:84 , srv1370997474:84 , srv1380217203:84 , srv1452822364:84 , srv1480533276:84 , srv1488188462:84 , srv1498783188:84 , srv1542829217:84 , srv1548541946:84 , srv1557685409:84 , srv1572612569:84 , srv1610261270:84 , srv1614757387:84 , srv1667676661:84 , srv1681696263:84 , srv1692772092:84 , srv1729278579:84 , srv1737127674:84 , srv173823246:84 , srv1746843677:84 , srv1749203440:84 , srv1750551797:84 , srv1812617073:84 , srv1823887586:84 , srv1866139205:84 , srv1936970975:84 , srv1968990935:84 , srv204115364:84 , srv2043499466:84 , srv206212508:84 , srv20650749:84 , srv2089211865:84 , srv2124629394:84 , srv2124960646:84 , srv2129948646:84 , srv218975516:84 , srv224321732:84 , srv227812429:84 , srv279579769:84 , srv299382910:84 , srv304515251:84 , srv316610983:84 , srv323770738:84 , srv347759344:84 , srv352739623:84 , srv391807523:84 , srv394322192:84 , srv404441971:84 , srv412844043:84 , srv458869014:84 , srv473761081:84 , srv478396568:84 , srv485938370:84 , srv490260069:84 , srv49267230:84 , srv519956734:84 , srv520115162:84 , srv523831004:84 , srv528908200:84 , srv545307566:84 , srv559548816:84 , srv565319268:84 , srv566724900:84 , srv585021229:84 , srv599422704:84 , srv607581903:84 , srv656081183:84 , srv67976789:84 , srv716751598:84 , srv716955675:84 , srv724776684:84 , srv739546504:84 , srv759143809:84 , srv765806536:84 , srv80391157:84 , srv816658746:84 , srv912175968:84 , srv92187752:84 , srv947315463:84 , srv952846153:84 , srv966822125:84 , srv787152573:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-23 00:00:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:22,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv478396568=69, srv1812617073=40, srv1667676661=31, srv519956734=73, srv545307566=77, srv607581903=83, srv656081183=84, srv1370997474=19, srv566724900=80, srv1183751389=12, srv49267230=72, srv1729278579=34, srv716751598=86, srv1557685409=27, srv1746843677=37, srv129552235=16, srv206212508=47, srv173823246=36, srv759143809=90, srv2124960646=51, srv2124629394=50, srv316610983=59, srv1347138386=17, srv1936970975=43, srv473761081=68, srv565319268=79, srv952846153=98, srv1081110801=3, srv1101862756=5, srv1233330773=14, srv765806536=91, srv218975516=53, srv2129948646=52, srv1257689904=15, srv204115364=45, srv599422704=82, srv1181206326=11, srv347759344=61, srv1614757387=30, srv391807523=63, srv520115162=74, srv1480533276=22, srv1087365607=4, srv1488188462=23, srv1750551797=39, srv227812429=55, srv912175968=95, srv20650749=48, srv352739623=62, srv394322192=64, srv1146911296=8, srv523831004=75, srv412844043=66, srv739546504=89, srv724776684=88, srv279579769=56, srv1572612569=28, srv92187752=96, srv1169127021=10, srv947315463=97, srv1692772092=33, srv1452822364=21, srv1498783188=24, srv1071314334=2, srv966822125=99, srv1360033009=18, srv1380217203=20, srv299382910=57, srv1737127674=35, srv1548541946=26, srv404441971=65, srv2089211865=49, srv224321732=54, srv80391157=93, srv1610261270=29, srv1681696263=32, srv1823887586=41, srv716955675=87, srv1145868826=7, srv816658746=94, srv1749203440=38, srv67976789=85, srv585021229=81, srv323770738=60, srv1053615599=0, srv1866139205=42, srv1054761085=1, srv1542829217=25, srv787152573=92, srv1968990935=44, srv304515251=58, srv490260069=71, srv528908200=76, srv1138436396=6, srv559548816=78, srv485938370=70, srv2043499466=46, srv1156922055=9, srv458869014=67, srv1207140819=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:22,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:22,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:22,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:22,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:22,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:22,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:22,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:22,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-23 00:00:22,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-23 00:00:22,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-23 00:00:22,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-23 00:00:22,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-23 00:00:22,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-23 00:00:22,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-23 00:00:22,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-23 00:00:22,871 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:22,871 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-23 00:00:22,871 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-23 00:00:23,218 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 597 ms to try 5000 different iterations. Found a solution that moves 4640 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18326726540886618. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038773759354949394, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053183712687005606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5155555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038773759354949394, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17891666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:23,232 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1557685409:80 , srv485938370:81 , srv759143809:81 , srv1169127021:82 , srv1488188462:82 , srv1614757387:82 , srv1749203440:82 , srv816658746:82 , srv1054761085:83 , srv1360033009:83 , srv1498783188:83 , srv1737127674:83 , srv716955675:83 , srv739546504:83 , srv1071314334:84 , srv1081110801:84 , srv1145868826:84 , srv1370997474:84 , srv173823246:84 , srv1936970975:84 , srv520115162:84 , srv566724900:84 , srv716751598:84 , srv724776684:84 , srv952846153:84 , srv966822125:84 , srv1138436396:85 , srv1233330773:85 , srv1548541946:85 , srv1866139205:85 , srv2124629394:85 , srv2124960646:85 , srv304515251:85 , srv316610983:85 , srv323770738:85 , srv404441971:85 , srv412844043:85 , srv519956734:85 , srv523831004:85 , srv607581903:85 , srv67976789:85 , srv92187752:85 , srv1101862756:86 , srv1181206326:86 , srv1667676661:86 , srv1692772092:86 , srv1746843677:86 , srv206212508:86 , srv20650749:86 , srv224321732:86 , srv347759344:86 , srv478396568:86 , srv528908200:86 , srv565319268:86 , srv656081183:86 , srv912175968:86 , srv129552235:87 , srv1380217203:87 , srv1480533276:87 , srv1542829217:87 , srv1729278579:87 , srv1968990935:87 , srv204115364:87 , srv2043499466:87 , srv2089211865:87 , srv352739623:87 , srv559548816:87 , srv765806536:87 , srv1257689904:88 , srv1347138386:88 , srv1572612569:88 , srv1610261270:88 , srv1750551797:88 , srv218975516:88 , srv227812429:88 , srv1053615599:89 , srv1156922055:89 , srv1823887586:89 , srv2129948646:89 , srv391807523:89 , srv599422704:89 , srv1087365607:90 , srv1146911296:90 , srv1207140819:90 , srv1681696263:90 , srv1812617073:90 , srv490260069:90 , srv279579769:91 , srv299382910:91 , srv458869014:91 , srv473761081:91 , srv49267230:91 , srv947315463:91 , srv1183751389:92 , srv545307566:92 , srv585021229:93 , srv80391157:93 , srv394322192:94 , srv1452822364:100 , srv787152573:434 } 2023-05-23 00:00:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:23,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv478396568=69, srv1812617073=40, srv1667676661=31, srv519956734=73, srv545307566=77, srv607581903=83, srv656081183=84, srv1370997474=19, srv566724900=80, srv1183751389=12, srv49267230=72, srv1729278579=34, srv716751598=86, srv1557685409=27, srv1746843677=37, srv129552235=16, srv206212508=47, srv173823246=36, srv759143809=90, srv2124960646=51, srv2124629394=50, srv316610983=59, srv1347138386=17, srv1936970975=43, srv473761081=68, srv565319268=79, srv952846153=98, srv1081110801=3, srv1101862756=5, srv1233330773=14, srv765806536=91, srv218975516=53, srv2129948646=52, srv1257689904=15, srv204115364=45, srv599422704=82, srv1181206326=11, srv347759344=61, srv1614757387=30, srv391807523=63, srv520115162=74, srv1480533276=22, srv1087365607=4, srv1488188462=23, srv1750551797=39, srv227812429=55, srv912175968=95, srv20650749=48, srv352739623=62, srv394322192=64, srv1146911296=8, srv523831004=75, srv412844043=66, srv739546504=89, srv724776684=88, srv279579769=56, srv1572612569=28, srv92187752=96, srv1169127021=10, srv947315463=97, srv1692772092=33, srv1452822364=21, srv1498783188=24, srv1071314334=2, srv966822125=99, srv1360033009=18, srv1380217203=20, srv299382910=57, srv1737127674=35, srv1548541946=26, srv404441971=65, srv2089211865=49, srv224321732=54, srv80391157=93, srv1610261270=29, srv1681696263=32, srv1823887586=41, srv716955675=87, srv1145868826=7, srv816658746=94, srv1749203440=38, srv67976789=85, srv585021229=81, srv323770738=60, srv1053615599=0, srv1866139205=42, srv1054761085=1, srv1542829217=25, srv787152573=92, srv1968990935=44, srv304515251=58, srv490260069=71, srv528908200=76, srv1138436396=6, srv559548816=78, srv485938370=70, srv2043499466=46, srv1156922055=9, srv458869014=67, srv1207140819=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:23,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:23,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:23,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:23,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-23 00:00:23,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-23 00:00:23,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-23 00:00:23,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-23 00:00:23,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-23 00:00:23,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-23 00:00:23,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-23 00:00:23,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-23 00:00:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-23 00:00:23,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-23 00:00:23,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-23 00:00:23,400 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038773759354949394, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053183712687005606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038773759354949394, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17891666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:23,400 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-23 00:00:23,400 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.183234775434444, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038773759354949394, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053183712687005606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038773759354949394, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17891666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-23 00:00:23,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 312 ms to try 5000 different iterations. Found a solution that moves 2639 regions; Going from a computed imbalance of 0.183234775434444 to a new imbalance of 0.01484391140311585. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021280846393994614); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03242534376177968, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2932222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021280846393994614); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:23,627 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1452822364:71 , srv394322192:78 , srv80391157:78 , srv458869014:79 , srv1207140819:80 , srv473761081:80 , srv585021229:80 , srv1183751389:81 , srv1681696263:81 , srv1823887586:81 , srv279579769:81 , srv49267230:81 , srv545307566:81 , srv947315463:81 , srv1812617073:82 , srv1053615599:83 , srv1087365607:83 , srv1146911296:83 , srv1610261270:83 , srv1968990935:83 , srv2129948646:83 , srv227812429:83 , srv391807523:83 , srv1156922055:84 , srv1257689904:84 , srv1572612569:84 , srv1750551797:84 , srv2089211865:84 , srv490260069:84 , srv559548816:84 , srv1138436396:85 , srv1233330773:85 , srv1866139205:85 , srv204115364:85 , srv224321732:85 , srv299382910:85 , srv316610983:85 , srv347759344:85 , srv478396568:85 , srv528908200:85 , srv599422704:85 , srv1101862756:86 , srv1181206326:86 , srv1729278579:86 , srv1746843677:86 , srv2043499466:86 , srv206212508:86 , srv2124960646:86 , srv218975516:86 , srv304515251:86 , srv352739623:86 , srv519956734:86 , srv565319268:86 , srv607581903:86 , srv67976789:86 , srv765806536:86 , srv1054761085:87 , srv1081110801:87 , srv129552235:87 , srv1347138386:87 , srv1360033009:87 , srv1370997474:87 , srv1380217203:87 , srv1480533276:87 , srv1488188462:87 , srv1498783188:87 , srv1548541946:87 , srv1667676661:87 , srv173823246:87 , srv20650749:87 , srv2124629394:87 , srv404441971:87 , srv656081183:87 , srv739546504:87 , srv92187752:87 , srv1071314334:88 , srv1614757387:88 , srv1692772092:88 , srv1737127674:88 , srv523831004:88 , srv816658746:88 , srv952846153:88 , srv1557685409:89 , srv1936970975:89 , srv323770738:89 , srv412844043:89 , srv566724900:89 , srv716751598:89 , srv912175968:89 , srv1145868826:90 , srv1542829217:90 , srv1749203440:90 , srv716955675:90 , srv724776684:90 , srv966822125:90 , srv1169127021:91 , srv520115162:91 , srv759143809:92 , srv485938370:93 , srv787152573:529 } 2023-05-23 00:00:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv478396568=69, srv1812617073=40, srv1667676661=31, srv519956734=73, srv545307566=77, srv607581903=83, srv656081183=84, srv1370997474=19, srv566724900=80, srv1183751389=12, srv49267230=72, srv1729278579=34, srv716751598=86, srv1557685409=27, srv1746843677=37, srv129552235=16, srv206212508=47, srv173823246=36, srv759143809=90, srv2124960646=51, srv2124629394=50, srv316610983=59, srv1347138386=17, srv1936970975=43, srv473761081=68, srv565319268=79, srv952846153=98, srv1081110801=3, srv1101862756=5, srv1233330773=14, srv765806536=91, srv218975516=53, srv2129948646=52, srv1257689904=15, srv204115364=45, srv599422704=82, srv1181206326=11, srv347759344=61, srv1614757387=30, srv391807523=63, srv520115162=74, srv1480533276=22, srv1087365607=4, srv1488188462=23, srv1750551797=39, srv227812429=55, srv912175968=95, srv20650749=48, srv352739623=62, srv394322192=64, srv1146911296=8, srv523831004=75, srv412844043=66, srv739546504=89, srv724776684=88, srv279579769=56, srv1572612569=28, srv92187752=96, srv1169127021=10, srv947315463=97, srv1692772092=33, srv1452822364=21, srv1498783188=24, srv1071314334=2, srv966822125=99, srv1360033009=18, srv1380217203=20, srv299382910=57, srv1737127674=35, srv1548541946=26, srv404441971=65, srv2089211865=49, srv224321732=54, srv80391157=93, srv1610261270=29, srv1681696263=32, srv1823887586=41, srv716955675=87, srv1145868826=7, srv816658746=94, srv1749203440=38, srv67976789=85, srv585021229=81, srv323770738=60, srv1053615599=0, srv1866139205=42, srv1054761085=1, srv1542829217=25, srv787152573=92, srv1968990935=44, srv304515251=58, srv490260069=71, srv528908200=76, srv1138436396=6, srv559548816=78, srv485938370=70, srv2043499466=46, srv1156922055=9, srv458869014=67, srv1207140819=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:23,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:23,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:23,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-23 00:00:23,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-23 00:00:23,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-23 00:00:23,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-23 00:00:23,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-23 00:00:23,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-23 00:00:23,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-23 00:00:23,745 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021280846393994614); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03242534376177968, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021280846393994614); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:23,745 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-23 00:00:23,745 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014825432730163252, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021280846393994614); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03242534376177968, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021280846393994614); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-23 00:00:23,905 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 235 ms to try 5000 different iterations. Found a solution that moves 979 regions; Going from a computed imbalance of 0.014825432730163252 to a new imbalance of 9.029912692139398E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004756182762796897); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.10877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004756182762796897); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:23,907 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1542829217:81 , srv299382910:81 , srv1347138386:82 , srv1380217203:83 , srv1480533276:83 , srv218975516:83 , srv520115162:83 , srv599422704:83 , srv912175968:83 , srv1145868826:84 , srv129552235:84 , srv1667676661:84 , srv1692772092:84 , srv1729278579:84 , srv1936970975:84 , srv20650749:84 , srv323770738:84 , srv412844043:84 , srv485938370:84 , srv523831004:84 , srv566724900:84 , srv724776684:84 , srv765806536:84 , srv966822125:84 , srv1087365607:85 , srv1101862756:85 , srv1146911296:85 , srv1156922055:85 , srv1169127021:85 , srv1183751389:85 , srv1548541946:85 , srv1746843677:85 , srv1749203440:85 , srv204115364:85 , srv2043499466:85 , srv2124629394:85 , srv352739623:85 , srv394322192:85 , srv545307566:85 , srv585021229:85 , srv656081183:85 , srv716751598:85 , srv759143809:85 , srv92187752:85 , srv947315463:85 , srv1053615599:86 , srv1071314334:86 , srv1081110801:86 , srv1181206326:86 , srv1257689904:86 , srv1370997474:86 , srv1452822364:86 , srv1572612569:86 , srv1610261270:86 , srv1681696263:86 , srv1750551797:86 , srv1812617073:86 , srv206212508:86 , srv2129948646:86 , srv224321732:86 , srv227812429:86 , srv279579769:86 , srv304515251:86 , srv347759344:86 , srv391807523:86 , srv404441971:86 , srv478396568:86 , srv490260069:86 , srv49267230:86 , srv519956734:86 , srv528908200:86 , srv559548816:86 , srv565319268:86 , srv716955675:86 , srv80391157:86 , srv952846153:86 , srv1054761085:87 , srv1138436396:87 , srv1207140819:87 , srv1614757387:87 , srv1737127674:87 , srv173823246:87 , srv1823887586:87 , srv1968990935:87 , srv2089211865:87 , srv2124960646:87 , srv458869014:87 , srv473761081:87 , srv607581903:87 , srv67976789:87 , srv739546504:87 , srv816658746:87 , srv1233330773:88 , srv1360033009:88 , srv1498783188:88 , srv1866139205:88 , srv316610983:88 , srv1488188462:89 , srv1557685409:89 , srv787152573:537 } 2023-05-23 00:00:23,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:23,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv478396568=69, srv1812617073=40, srv1667676661=31, srv519956734=73, srv545307566=77, srv607581903=83, srv656081183=84, srv1370997474=19, srv566724900=80, srv1183751389=12, srv49267230=72, srv1729278579=34, srv716751598=86, srv1557685409=27, srv1746843677=37, srv129552235=16, srv206212508=47, srv173823246=36, srv759143809=90, srv2124960646=51, srv2124629394=50, srv316610983=59, srv1347138386=17, srv1936970975=43, srv473761081=68, srv565319268=79, srv952846153=98, srv1081110801=3, srv1101862756=5, srv1233330773=14, srv765806536=91, srv218975516=53, srv2129948646=52, srv1257689904=15, srv204115364=45, srv599422704=82, srv1181206326=11, srv347759344=61, srv1614757387=30, srv391807523=63, srv520115162=74, srv1480533276=22, srv1087365607=4, srv1488188462=23, srv1750551797=39, srv227812429=55, srv912175968=95, srv20650749=48, srv352739623=62, srv394322192=64, srv1146911296=8, srv523831004=75, srv412844043=66, srv739546504=89, srv724776684=88, srv279579769=56, srv1572612569=28, srv92187752=96, srv1169127021=10, srv947315463=97, srv1692772092=33, srv1452822364=21, srv1498783188=24, srv1071314334=2, srv966822125=99, srv1360033009=18, srv1380217203=20, srv299382910=57, srv1737127674=35, srv1548541946=26, srv404441971=65, srv2089211865=49, srv224321732=54, srv80391157=93, srv1610261270=29, srv1681696263=32, srv1823887586=41, srv716955675=87, srv1145868826=7, srv816658746=94, srv1749203440=38, srv67976789=85, srv585021229=81, srv323770738=60, srv1053615599=0, srv1866139205=42, srv1054761085=1, srv1542829217=25, srv787152573=92, srv1968990935=44, srv304515251=58, srv490260069=71, srv528908200=76, srv1138436396=6, srv559548816=78, srv485938370=70, srv2043499466=46, srv1156922055=9, srv458869014=67, srv1207140819=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-23 00:00:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-23 00:00:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-23 00:00:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-23 00:00:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-23 00:00:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-23 00:00:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-23 00:00:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-23 00:00:24,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004756182762796897); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004756182762796897); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:24,011 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-23 00:00:24,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.961361646968531E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004756182762796897); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004756182762796897); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-23 00:00:24,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations. Found a solution that moves 265 regions; Going from a computed imbalance of 8.961361646968531E-4 to a new imbalance of 2.7072386556763017E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.029444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:24,225 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv490260069:82 , srv1183751389:83 , srv1498783188:83 , srv1812617073:83 , srv716955675:83 , srv1053615599:84 , srv1071314334:84 , srv1087365607:84 , srv1145868826:84 , srv1146911296:84 , srv1169127021:84 , srv1181206326:84 , srv1233330773:84 , srv1257689904:84 , srv129552235:84 , srv1370997474:84 , srv1452822364:84 , srv1488188462:84 , srv1542829217:84 , srv1557685409:84 , srv1572612569:84 , srv1610261270:84 , srv1692772092:84 , srv1729278579:84 , srv1737127674:84 , srv173823246:84 , srv1866139205:84 , srv20650749:84 , srv2089211865:84 , srv2124960646:84 , srv2129948646:84 , srv279579769:84 , srv316610983:84 , srv323770738:84 , srv352739623:84 , srv391807523:84 , srv412844043:84 , srv473761081:84 , srv478396568:84 , srv485938370:84 , srv49267230:84 , srv520115162:84 , srv545307566:84 , srv565319268:84 , srv585021229:84 , srv607581903:84 , srv656081183:84 , srv67976789:84 , srv716751598:84 , srv724776684:84 , srv759143809:84 , srv92187752:84 , srv952846153:84 , srv966822125:84 , srv1054761085:85 , srv1081110801:85 , srv1138436396:85 , srv1156922055:85 , srv1207140819:85 , srv1347138386:85 , srv1360033009:85 , srv1380217203:85 , srv1480533276:85 , srv1548541946:85 , srv1614757387:85 , srv1681696263:85 , srv1746843677:85 , srv1749203440:85 , srv1750551797:85 , srv1823887586:85 , srv1936970975:85 , srv1968990935:85 , srv204115364:85 , srv2043499466:85 , srv206212508:85 , srv2124629394:85 , srv218975516:85 , srv224321732:85 , srv227812429:85 , srv299382910:85 , srv304515251:85 , srv347759344:85 , srv394322192:85 , srv404441971:85 , srv458869014:85 , srv519956734:85 , srv528908200:85 , srv559548816:85 , srv566724900:85 , srv599422704:85 , srv739546504:85 , srv80391157:85 , srv816658746:85 , srv912175968:85 , srv947315463:85 , srv1101862756:86 , srv1667676661:86 , srv523831004:86 , srv765806536:86 , srv787152573:641 } 2023-05-23 00:00:24,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:24,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv478396568=69, srv1812617073=40, srv1667676661=31, srv519956734=73, srv545307566=77, srv607581903=83, srv656081183=84, srv1370997474=19, srv566724900=80, srv1183751389=12, srv49267230=72, srv1729278579=34, srv716751598=86, srv1557685409=27, srv1746843677=37, srv129552235=16, srv206212508=47, srv173823246=36, srv759143809=90, srv2124960646=51, srv2124629394=50, srv316610983=59, srv1347138386=17, srv1936970975=43, srv473761081=68, srv565319268=79, srv952846153=98, srv1081110801=3, srv1101862756=5, srv1233330773=14, srv765806536=91, srv218975516=53, srv2129948646=52, srv1257689904=15, srv204115364=45, srv599422704=82, srv1181206326=11, srv347759344=61, srv1614757387=30, srv391807523=63, srv520115162=74, srv1480533276=22, srv1087365607=4, srv1488188462=23, srv1750551797=39, srv227812429=55, srv912175968=95, srv20650749=48, srv352739623=62, srv394322192=64, srv1146911296=8, srv523831004=75, srv412844043=66, srv739546504=89, srv724776684=88, srv279579769=56, srv1572612569=28, srv92187752=96, srv1169127021=10, srv947315463=97, srv1692772092=33, srv1452822364=21, srv1498783188=24, srv1071314334=2, srv966822125=99, srv1360033009=18, srv1380217203=20, srv299382910=57, srv1737127674=35, srv1548541946=26, srv404441971=65, srv2089211865=49, srv224321732=54, srv80391157=93, srv1610261270=29, srv1681696263=32, srv1823887586=41, srv716955675=87, srv1145868826=7, srv816658746=94, srv1749203440=38, srv67976789=85, srv585021229=81, srv323770738=60, srv1053615599=0, srv1866139205=42, srv1054761085=1, srv1542829217=25, srv787152573=92, srv1968990935=44, srv304515251=58, srv490260069=71, srv528908200=76, srv1138436396=6, srv559548816=78, srv485938370=70, srv2043499466=46, srv1156922055=9, srv458869014=67, srv1207140819=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:24,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:24,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:24,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-23 00:00:24,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-23 00:00:24,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-23 00:00:24,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-23 00:00:24,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-23 00:00:24,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-23 00:00:24,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-23 00:00:24,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-23 00:00:24,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-23 00:00:24,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-23 00:00:24,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-23 00:00:24,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-23 00:00:24,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-23 00:00:24,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-23 00:00:24,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-23 00:00:24,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:24,335 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-23 00:00:24,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.5216816896877434E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-23 00:00:24,473 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 213 ms to try 5000 different iterations. Found a solution that moves 21 regions; Going from a computed imbalance of 2.5216816896877434E-5 to a new imbalance of 6.8911141966908056E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0023333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:24,473 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1101862756:83 , srv1156922055:83 , srv1360033009:83 , srv1667676661:83 , srv1750551797:83 , srv2043499466:83 , srv206212508:83 , srv404441971:83 , srv523831004:83 , srv765806536:83 , srv912175968:83 , srv1053615599:84 , srv1054761085:84 , srv1071314334:84 , srv1081110801:84 , srv1087365607:84 , srv1138436396:84 , srv1145868826:84 , srv1146911296:84 , srv1169127021:84 , srv1181206326:84 , srv1207140819:84 , srv1233330773:84 , srv1257689904:84 , srv129552235:84 , srv1347138386:84 , srv1380217203:84 , srv1480533276:84 , srv1488188462:84 , srv1542829217:84 , srv1548541946:84 , srv1557685409:84 , srv1572612569:84 , srv1614757387:84 , srv1681696263:84 , srv1692772092:84 , srv1737127674:84 , srv173823246:84 , srv1746843677:84 , srv1749203440:84 , srv1823887586:84 , srv1866139205:84 , srv1936970975:84 , srv1968990935:84 , srv204115364:84 , srv2089211865:84 , srv2124629394:84 , srv2124960646:84 , srv2129948646:84 , srv218975516:84 , srv224321732:84 , srv227812429:84 , srv279579769:84 , srv299382910:84 , srv304515251:84 , srv316610983:84 , srv323770738:84 , srv347759344:84 , srv352739623:84 , srv391807523:84 , srv394322192:84 , srv412844043:84 , srv458869014:84 , srv473761081:84 , srv485938370:84 , srv490260069:84 , srv49267230:84 , srv519956734:84 , srv520115162:84 , srv528908200:84 , srv545307566:84 , srv559548816:84 , srv565319268:84 , srv566724900:84 , srv585021229:84 , srv599422704:84 , srv607581903:84 , srv656081183:84 , srv67976789:84 , srv716751598:84 , srv716955675:84 , srv724776684:84 , srv739546504:84 , srv759143809:84 , srv80391157:84 , srv816658746:84 , srv947315463:84 , srv952846153:84 , srv966822125:84 , srv1183751389:85 , srv1370997474:85 , srv1452822364:85 , srv1498783188:85 , srv1610261270:85 , srv1729278579:85 , srv1812617073:85 , srv20650749:85 , srv478396568:85 , srv92187752:85 , srv787152573:685 } 2023-05-23 00:00:24,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:24,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv478396568=69, srv1812617073=40, srv1667676661=31, srv519956734=73, srv545307566=77, srv607581903=83, srv656081183=84, srv1370997474=19, srv566724900=80, srv1183751389=12, srv49267230=72, srv1729278579=34, srv716751598=86, srv1557685409=27, srv1746843677=37, srv129552235=16, srv206212508=47, srv173823246=36, srv759143809=90, srv2124960646=51, srv2124629394=50, srv316610983=59, srv1347138386=17, srv1936970975=43, srv473761081=68, srv565319268=79, srv952846153=98, srv1081110801=3, srv1101862756=5, srv1233330773=14, srv765806536=91, srv218975516=53, srv2129948646=52, srv1257689904=15, srv204115364=45, srv599422704=82, srv1181206326=11, srv347759344=61, srv1614757387=30, srv391807523=63, srv520115162=74, srv1480533276=22, srv1087365607=4, srv1488188462=23, srv1750551797=39, srv227812429=55, srv912175968=95, srv20650749=48, srv352739623=62, srv394322192=64, srv1146911296=8, srv523831004=75, srv412844043=66, srv739546504=89, srv724776684=88, srv279579769=56, srv1572612569=28, srv92187752=96, srv1169127021=10, srv947315463=97, srv1692772092=33, srv1452822364=21, srv1498783188=24, srv1071314334=2, srv966822125=99, srv1360033009=18, srv1380217203=20, srv299382910=57, srv1737127674=35, srv1548541946=26, srv404441971=65, srv2089211865=49, srv224321732=54, srv80391157=93, srv1610261270=29, srv1681696263=32, srv1823887586=41, srv716955675=87, srv1145868826=7, srv816658746=94, srv1749203440=38, srv67976789=85, srv585021229=81, srv323770738=60, srv1053615599=0, srv1866139205=42, srv1054761085=1, srv1542829217=25, srv787152573=92, srv1968990935=44, srv304515251=58, srv490260069=71, srv528908200=76, srv1138436396=6, srv559548816=78, srv485938370=70, srv2043499466=46, srv1156922055=9, srv458869014=67, srv1207140819=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:24,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-23 00:00:24,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-23 00:00:24,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-23 00:00:24,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-23 00:00:24,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-23 00:00:24,566 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:24,567 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1101862756:83 , srv1156922055:83 , srv1360033009:83 , srv1667676661:83 , srv1750551797:83 , srv2043499466:83 , srv206212508:83 , srv404441971:83 , srv523831004:83 , srv765806536:83 , srv912175968:83 , srv1053615599:84 , srv1054761085:84 , srv1071314334:84 , srv1081110801:84 , srv1087365607:84 , srv1138436396:84 , srv1145868826:84 , srv1146911296:84 , srv1169127021:84 , srv1181206326:84 , srv1207140819:84 , srv1233330773:84 , srv1257689904:84 , srv129552235:84 , srv1347138386:84 , srv1380217203:84 , srv1480533276:84 , srv1488188462:84 , srv1542829217:84 , srv1548541946:84 , srv1557685409:84 , srv1572612569:84 , srv1614757387:84 , srv1681696263:84 , srv1692772092:84 , srv1737127674:84 , srv173823246:84 , srv1746843677:84 , srv1749203440:84 , srv1823887586:84 , srv1866139205:84 , srv1936970975:84 , srv1968990935:84 , srv204115364:84 , srv2089211865:84 , srv2124629394:84 , srv2124960646:84 , srv2129948646:84 , srv218975516:84 , srv224321732:84 , srv227812429:84 , srv279579769:84 , srv299382910:84 , srv304515251:84 , srv316610983:84 , srv323770738:84 , srv347759344:84 , srv352739623:84 , srv391807523:84 , srv394322192:84 , srv412844043:84 , srv458869014:84 , srv473761081:84 , srv485938370:84 , srv490260069:84 , srv49267230:84 , srv519956734:84 , srv520115162:84 , srv528908200:84 , srv545307566:84 , srv559548816:84 , srv565319268:84 , srv566724900:84 , srv585021229:84 , srv599422704:84 , srv607581903:84 , srv656081183:84 , srv67976789:84 , srv716751598:84 , srv716955675:84 , srv724776684:84 , srv739546504:84 , srv759143809:84 , srv80391157:84 , srv816658746:84 , srv947315463:84 , srv952846153:84 , srv966822125:84 , srv1183751389:85 , srv1370997474:85 , srv1452822364:85 , srv1498783188:85 , srv1610261270:85 , srv1729278579:85 , srv1812617073:85 , srv20650749:85 , srv478396568:85 , srv92187752:85 , srv787152573:685 } 2023-05-23 00:00:24,590 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=121 (was 121), ProcessCount=170 (was 170), AvailableMemoryMB=7663 (was 8156) 2023-05-23 00:00:24,596 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=121, ProcessCount=170, AvailableMemoryMB=7663 2023-05-23 00:00:24,597 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-23 00:00:24,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-23 00:00:24,609 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1112502657:84 , srv1171237733:84 , srv1193295707:84 , srv1442514017:84 , srv1466702452:84 , srv1513836089:84 , srv1524116061:84 , srv1624982106:84 , srv165574402:84 , srv1896618298:84 , srv1955729261:84 , srv2083313956:84 , srv249844053:84 , srv331828432:84 , srv338999716:84 , srv355084645:84 , srv38078084:84 , srv408761433:84 , srv48606887:84 , srv580735293:84 , srv615087956:84 , srv69022538:84 , srv707941894:84 , srv746871594:84 , srv764395245:84 , srv774194730:84 , srv874265442:84 , srv894503756:84 , srv950819339:84 , srv1390216361:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-23 00:00:24,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:00:24,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1896618298=10, srv1171237733=1, srv1955729261=11, srv615087956=21, srv1624982106=8, srv746871594=24, srv48606887=19, srv1442514017=4, srv2083313956=12, srv249844053=13, srv894503756=28, srv950819339=29, srv1390216361=3, srv69022538=22, srv764395245=25, srv707941894=23, srv874265442=27, srv408761433=18, srv38078084=17, srv1466702452=5, srv165574402=9, srv580735293=20, srv1524116061=7, srv774194730=26, srv331828432=14, srv1193295707=2, srv338999716=15, srv355084645=16, srv1513836089=6, srv1112502657=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:00:24,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:00:24,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:00:24,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-23 00:00:24,649 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.085798554355061, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:00:24,649 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912463391707472, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.085798554355061, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-23 00:02:24,630 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 10093966 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912463391707472 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:02:24,635 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1112502657:90 , srv1171237733:90 , srv1193295707:90 , srv1390216361:90 , srv1442514017:90 , srv1466702452:90 , srv1513836089:90 , srv1524116061:90 , srv1624982106:90 , srv165574402:90 , srv1896618298:90 , srv1955729261:90 , srv2083313956:90 , srv249844053:90 , srv331828432:90 , srv338999716:90 , srv355084645:90 , srv38078084:90 , srv408761433:90 , srv48606887:90 , srv580735293:90 , srv615087956:90 , srv69022538:90 , srv707941894:90 , srv746871594:90 , srv764395245:90 , srv774194730:90 , srv874265442:90 , srv894503756:90 , srv950819339:90 } 2023-05-23 00:02:24,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-23 00:02:24,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1896618298=10, srv1171237733=1, srv1955729261=11, srv615087956=21, srv1624982106=8, srv746871594=24, srv48606887=19, srv1442514017=4, srv2083313956=12, srv249844053=13, srv894503756=28, srv950819339=29, srv1390216361=3, srv69022538=22, srv764395245=25, srv707941894=23, srv874265442=27, srv408761433=18, srv38078084=17, srv1466702452=5, srv165574402=9, srv580735293=20, srv1524116061=7, srv774194730=26, srv331828432=14, srv1193295707=2, srv338999716=15, srv355084645=16, srv1513836089=6, srv1112502657=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-23 00:02:24,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-23 00:02:24,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-23 00:02:24,650 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-23 00:02:24,650 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1112502657:90 , srv1171237733:90 , srv1193295707:90 , srv1390216361:90 , srv1442514017:90 , srv1466702452:90 , srv1513836089:90 , srv1524116061:90 , srv1624982106:90 , srv165574402:90 , srv1896618298:90 , srv1955729261:90 , srv2083313956:90 , srv249844053:90 , srv331828432:90 , srv338999716:90 , srv355084645:90 , srv38078084:90 , srv408761433:90 , srv48606887:90 , srv580735293:90 , srv615087956:90 , srv69022538:90 , srv707941894:90 , srv746871594:90 , srv764395245:90 , srv774194730:90 , srv874265442:90 , srv894503756:90 , srv950819339:90 } 2023-05-23 00:02:24,660 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=133 (was 121) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 170), AvailableMemoryMB=7538 (was 7663)