2023-02-28 14:55:40,899 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-28 14:55:41,185 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-28 14:55:41,320 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-28 14:55:41,402 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-28 14:55:41,411 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-28 14:55:41,427 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=267, ProcessCount=165, AvailableMemoryMB=5366 2023-02-28 14:55:41,429 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-28 14:55:41,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-28 14:55:41,585 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011265298:84 , srv1019806956:84 , srv1023332809:84 , srv1031145770:84 , srv1042623447:84 , srv1044106307:84 , srv1058668961:84 , srv1064576425:84 , srv1067872885:84 , srv1088525795:84 , srv1090743316:84 , srv1144880579:84 , srv1188631627:84 , srv1193993359:84 , srv1203150557:84 , srv1205202746:84 , srv1236163417:84 , srv1305030422:84 , srv135480642:84 , srv1367499866:84 , srv1392936988:84 , srv139363433:84 , srv1395555021:84 , srv1399951906:84 , srv141197585:84 , srv1412119170:84 , srv1432022010:84 , srv1442388113:84 , srv1456771847:84 , srv1493047310:84 , srv1512366016:84 , srv1513519482:84 , srv1524853683:84 , srv1532019594:84 , srv1535632161:84 , srv1556302173:84 , srv1566597924:84 , srv1610055821:84 , srv1671217143:84 , srv1711766728:84 , srv1742342609:84 , srv1773282907:84 , srv1794447563:84 , srv1797431387:84 , srv1832871775:84 , srv1857736799:84 , srv1921653455:84 , srv1926661269:84 , srv1958748404:84 , srv2006886002:84 , srv2022548994:84 , srv20377607:84 , srv2063601679:84 , srv2081401803:84 , srv2109864750:84 , srv2121120930:84 , srv2125727028:84 , srv213993090:84 , srv255937429:84 , srv258801848:84 , srv272000377:84 , srv27380108:84 , srv280752567:84 , srv295691195:84 , srv31223744:84 , srv329870529:84 , srv358894402:84 , srv401232733:84 , srv440819952:84 , srv447668814:84 , srv458009397:84 , srv458409058:84 , srv476992859:84 , srv516711512:84 , srv526259885:84 , srv526571057:84 , srv535664243:84 , srv553913771:84 , srv569339610:84 , srv649508455:84 , srv66694292:84 , srv692177885:84 , srv695788418:84 , srv698002097:84 , srv70228209:84 , srv722530749:84 , srv735784904:84 , srv741718104:84 , srv834081746:84 , srv838772264:84 , srv845051879:84 , srv869686140:84 , srv877338351:84 , srv886431189:84 , srv913163185:84 , srv929746468:84 , srv939883570:84 , srv953963333:84 , srv968400860:84 , srv712899409:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-28 14:55:41,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272000377=60, srv877338351=93, srv526259885=74, srv1023332809=2, srv838772264=90, srv953963333=98, srv280752567=62, srv1532019594=33, srv1031145770=3, srv141197585=24, srv1412119170=25, srv1193993359=13, srv2006886002=49, srv1203150557=14, srv913163185=95, srv458409058=71, srv458009397=70, srv1512366016=30, srv1566597924=36, srv1671217143=38, srv2063601679=52, srv1513519482=31, srv213993090=57, srv258801848=59, srv1042623447=4, srv929746468=96, srv869686140=92, srv886431189=94, srv939883570=97, srv2022548994=50, srv1442388113=27, srv516711512=73, srv535664243=76, srv1742342609=40, srv401232733=67, srv1958748404=48, srv358894402=66, srv440819952=68, srv70228209=84, srv2121120930=55, srv1832871775=44, srv1926661269=47, srv1090743316=10, srv1794447563=42, srv735784904=87, srv1067872885=8, srv20377607=51, srv712899409=85, srv845051879=91, srv695788418=82, srv1395555021=22, srv1456771847=28, srv1367499866=19, srv1399951906=23, srv66694292=80, srv135480642=18, srv649508455=79, srv1188631627=12, srv27380108=61, srv1058668961=6, srv1236163417=16, srv447668814=69, srv476992859=72, srv1392936988=20, srv1535632161=34, srv2125727028=56, srv1088525795=9, srv1711766728=39, srv569339610=78, srv329870529=65, srv1305030422=17, srv553913771=77, srv139363433=21, srv1064576425=7, srv968400860=99, srv722530749=86, srv1432022010=26, srv1144880579=11, srv1857736799=45, srv1921653455=46, srv295691195=63, srv526571057=75, srv1493047310=29, srv1797431387=43, srv698002097=83, srv2081401803=53, srv1524853683=32, srv1044106307=5, srv741718104=88, srv1205202746=15, srv834081746=89, srv1011265298=0, srv255937429=58, srv1773282907=41, srv31223744=64, srv1610055821=37, srv2109864750=54, srv1019806956=1, srv1556302173=35, srv692177885=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:41,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:41,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:41,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:41,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:41,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:41,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:41,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:41,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:41,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:41,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:41,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:41,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:41,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:41,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:41,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:41,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:41,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:41,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-28 14:55:41,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-28 14:55:41,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-28 14:55:41,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-28 14:55:41,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-28 14:55:41,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-28 14:55:41,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-28 14:55:41,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-28 14:55:41,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-28 14:55:41,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-28 14:55:41,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-28 14:55:41,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-28 14:55:41,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-28 14:55:41,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-28 14:55:41,992 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:41,993 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-28 14:55:41,993 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-28 14:55:42,375 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 597 ms to try 5000 different iterations. Found a solution that moves 4734 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17609631197848263. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04133904115640732, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06223665056380102, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.526, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04133904115640732, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24116666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:42,420 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1019806956:78 , srv1921653455:80 , srv649508455:80 , srv2121120930:81 , srv1794447563:82 , srv27380108:82 , srv358894402:82 , srv722530749:82 , srv913163185:82 , srv1042623447:83 , srv1367499866:83 , srv1432022010:83 , srv1556302173:83 , srv2125727028:83 , srv255937429:83 , srv295691195:83 , srv1023332809:84 , srv1064576425:84 , srv1144880579:84 , srv1188631627:84 , srv1305030422:84 , srv1392936988:84 , srv1395555021:84 , srv1535632161:84 , srv1742342609:84 , srv2022548994:84 , srv458409058:84 , srv526571057:84 , srv569339610:84 , srv869686140:84 , srv1205202746:85 , srv1566597924:85 , srv2109864750:85 , srv516711512:85 , srv66694292:85 , srv692177885:85 , srv698002097:85 , srv886431189:85 , srv929746468:85 , srv1067872885:86 , srv1193993359:86 , srv1442388113:86 , srv1513519482:86 , srv1773282907:86 , srv1797431387:86 , srv31223744:86 , srv440819952:86 , srv535664243:86 , srv735784904:86 , srv741718104:86 , srv953963333:86 , srv1031145770:87 , srv1090743316:87 , srv135480642:87 , srv1456771847:87 , srv1524853683:87 , srv1610055821:87 , srv1857736799:87 , srv1958748404:87 , srv2006886002:87 , srv20377607:87 , srv213993090:87 , srv258801848:87 , srv272000377:87 , srv280752567:87 , srv401232733:87 , srv877338351:87 , srv968400860:87 , srv1058668961:88 , srv1399951906:88 , srv141197585:88 , srv1493047310:88 , srv1711766728:88 , srv1832871775:88 , srv1926661269:88 , srv2063601679:88 , srv2081401803:88 , srv458009397:88 , srv476992859:88 , srv526259885:88 , srv695788418:88 , srv845051879:88 , srv939883570:88 , srv1044106307:89 , srv1236163417:89 , srv1671217143:89 , srv70228209:89 , srv838772264:89 , srv1011265298:90 , srv1088525795:90 , srv139363433:90 , srv1512366016:90 , srv553913771:90 , srv447668814:93 , srv1203150557:94 , srv1412119170:94 , srv1532019594:94 , srv834081746:94 , srv329870529:97 , srv712899409:457 } 2023-02-28 14:55:42,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:42,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272000377=60, srv877338351=93, srv526259885=74, srv1023332809=2, srv838772264=90, srv953963333=98, srv280752567=62, srv1532019594=33, srv1031145770=3, srv141197585=24, srv1412119170=25, srv1193993359=13, srv2006886002=49, srv1203150557=14, srv913163185=95, srv458409058=71, srv458009397=70, srv1512366016=30, srv1566597924=36, srv1671217143=38, srv2063601679=52, srv1513519482=31, srv213993090=57, srv258801848=59, srv1042623447=4, srv929746468=96, srv869686140=92, srv886431189=94, srv939883570=97, srv2022548994=50, srv1442388113=27, srv516711512=73, srv535664243=76, srv1742342609=40, srv401232733=67, srv1958748404=48, srv358894402=66, srv440819952=68, srv70228209=84, srv2121120930=55, srv1832871775=44, srv1926661269=47, srv1090743316=10, srv1794447563=42, srv735784904=87, srv1067872885=8, srv20377607=51, srv712899409=85, srv845051879=91, srv695788418=82, srv1395555021=22, srv1456771847=28, srv1367499866=19, srv1399951906=23, srv66694292=80, srv135480642=18, srv649508455=79, srv1188631627=12, srv27380108=61, srv1058668961=6, srv1236163417=16, srv447668814=69, srv476992859=72, srv1392936988=20, srv1535632161=34, srv2125727028=56, srv1088525795=9, srv1711766728=39, srv569339610=78, srv329870529=65, srv1305030422=17, srv553913771=77, srv139363433=21, srv1064576425=7, srv968400860=99, srv722530749=86, srv1432022010=26, srv1144880579=11, srv1857736799=45, srv1921653455=46, srv295691195=63, srv526571057=75, srv1493047310=29, srv1797431387=43, srv698002097=83, srv2081401803=53, srv1524853683=32, srv1044106307=5, srv741718104=88, srv1205202746=15, srv834081746=89, srv1011265298=0, srv255937429=58, srv1773282907=41, srv31223744=64, srv1610055821=37, srv2109864750=54, srv1019806956=1, srv1556302173=35, srv692177885=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:42,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:42,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:42,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:42,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:42,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:42,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:42,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:42,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:42,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:42,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:42,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:42,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-28 14:55:42,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-28 14:55:42,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-28 14:55:42,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-28 14:55:42,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-28 14:55:42,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-28 14:55:42,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-28 14:55:42,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-28 14:55:42,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-28 14:55:42,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-28 14:55:42,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-28 14:55:42,685 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04133904115640732, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06223665056380102, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04133904115640732, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24116666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:42,686 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-28 14:55:42,686 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17606316380199244, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04133904115640732, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06223665056380102, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04133904115640732, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24116666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-28 14:55:43,027 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 514 ms to try 5000 different iterations. Found a solution that moves 2650 regions; Going from a computed imbalance of 0.17606316380199244 to a new imbalance of 0.015274349665732054. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.025557263398747754, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04178938158215648, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.025557263398747754, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:43,038 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv329870529:77 , srv834081746:78 , srv1412119170:79 , srv1532019594:79 , srv447668814:79 , srv1011265298:80 , srv1203150557:81 , srv1236163417:81 , srv1512366016:81 , srv526259885:81 , srv1088525795:82 , srv1493047310:82 , srv1524853683:82 , srv1671217143:82 , srv1958748404:82 , srv2081401803:82 , srv695788418:82 , srv845051879:82 , srv939883570:82 , srv1044106307:83 , srv1090743316:83 , srv141197585:83 , srv1456771847:83 , srv1832871775:83 , srv2063601679:83 , srv258801848:83 , srv70228209:83 , srv968400860:83 , srv1797431387:84 , srv2109864750:84 , srv31223744:84 , srv458009397:84 , srv553913771:84 , srv1023332809:85 , srv1031145770:85 , srv1067872885:85 , srv1442388113:85 , srv1513519482:85 , srv1857736799:85 , srv1926661269:85 , srv20377607:85 , srv213993090:85 , srv516711512:85 , srv535664243:85 , srv66694292:85 , srv735784904:85 , srv1064576425:86 , srv135480642:86 , srv139363433:86 , srv1399951906:86 , srv1566597924:86 , srv1610055821:86 , srv1711766728:86 , srv1742342609:86 , srv2006886002:86 , srv272000377:86 , srv280752567:86 , srv401232733:86 , srv440819952:86 , srv458409058:86 , srv741718104:86 , srv886431189:86 , srv1144880579:87 , srv1395555021:87 , srv1432022010:87 , srv2125727028:87 , srv358894402:87 , srv476992859:87 , srv722530749:87 , srv877338351:87 , srv953963333:87 , srv1188631627:88 , srv1205202746:88 , srv1305030422:88 , srv1535632161:88 , srv1773282907:88 , srv255937429:88 , srv27380108:88 , srv295691195:88 , srv569339610:88 , srv692177885:88 , srv698002097:88 , srv838772264:88 , srv869686140:88 , srv1042623447:89 , srv1193993359:89 , srv1367499866:89 , srv1392936988:89 , srv1556302173:89 , srv2022548994:89 , srv2121120930:89 , srv913163185:89 , srv929746468:89 , srv1794447563:90 , srv526571057:90 , srv1058668961:91 , srv1921653455:91 , srv1019806956:92 , srv649508455:92 , srv712899409:544 } 2023-02-28 14:55:43,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:43,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272000377=60, srv877338351=93, srv526259885=74, srv1023332809=2, srv838772264=90, srv953963333=98, srv280752567=62, srv1532019594=33, srv1031145770=3, srv141197585=24, srv1412119170=25, srv1193993359=13, srv2006886002=49, srv1203150557=14, srv913163185=95, srv458409058=71, srv458009397=70, srv1512366016=30, srv1566597924=36, srv1671217143=38, srv2063601679=52, srv1513519482=31, srv213993090=57, srv258801848=59, srv1042623447=4, srv929746468=96, srv869686140=92, srv886431189=94, srv939883570=97, srv2022548994=50, srv1442388113=27, srv516711512=73, srv535664243=76, srv1742342609=40, srv401232733=67, srv1958748404=48, srv358894402=66, srv440819952=68, srv70228209=84, srv2121120930=55, srv1832871775=44, srv1926661269=47, srv1090743316=10, srv1794447563=42, srv735784904=87, srv1067872885=8, srv20377607=51, srv712899409=85, srv845051879=91, srv695788418=82, srv1395555021=22, srv1456771847=28, srv1367499866=19, srv1399951906=23, srv66694292=80, srv135480642=18, srv649508455=79, srv1188631627=12, srv27380108=61, srv1058668961=6, srv1236163417=16, srv447668814=69, srv476992859=72, srv1392936988=20, srv1535632161=34, srv2125727028=56, srv1088525795=9, srv1711766728=39, srv569339610=78, srv329870529=65, srv1305030422=17, srv553913771=77, srv139363433=21, srv1064576425=7, srv968400860=99, srv722530749=86, srv1432022010=26, srv1144880579=11, srv1857736799=45, srv1921653455=46, srv295691195=63, srv526571057=75, srv1493047310=29, srv1797431387=43, srv698002097=83, srv2081401803=53, srv1524853683=32, srv1044106307=5, srv741718104=88, srv1205202746=15, srv834081746=89, srv1011265298=0, srv255937429=58, srv1773282907=41, srv31223744=64, srv1610055821=37, srv2109864750=54, srv1019806956=1, srv1556302173=35, srv692177885=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-28 14:55:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-28 14:55:43,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-28 14:55:43,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-28 14:55:43,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-28 14:55:43,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-28 14:55:43,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-28 14:55:43,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-28 14:55:43,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-28 14:55:43,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-28 14:55:43,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-28 14:55:43,184 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.025557263398747754, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04178938158215648, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.025557263398747754, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:43,185 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-28 14:55:43,185 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0152557939691332, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.025557263398747754, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04178938158215648, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.025557263398747754, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-28 14:55:43,405 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 311 ms to try 5000 different iterations. Found a solution that moves 989 regions; Going from a computed imbalance of 0.0152557939691332 to a new imbalance of 9.274468333494852E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008469335398956192); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.017102057161260623); MoveCostFunction : (multiplier=7.0, imbalance=0.10988888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008469335398956192); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:43,407 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1058668961:79 , srv838772264:81 , srv1193993359:82 , srv139363433:82 , srv877338351:82 , srv1203150557:83 , srv2006886002:83 , srv476992859:83 , srv929746468:83 , srv953963333:83 , srv1188631627:84 , srv1205202746:84 , srv135480642:84 , srv1392936988:84 , srv1399951906:84 , srv1412119170:84 , srv1532019594:84 , srv1556302173:84 , srv1711766728:84 , srv1773282907:84 , srv1926661269:84 , srv272000377:84 , srv440819952:84 , srv526571057:84 , srv553913771:84 , srv70228209:84 , srv869686140:84 , srv1031145770:85 , srv1042623447:85 , srv1044106307:85 , srv1088525795:85 , srv1144880579:85 , srv1367499866:85 , srv141197585:85 , srv1610055821:85 , srv1794447563:85 , srv1857736799:85 , srv1921653455:85 , srv213993090:85 , srv280752567:85 , srv329870529:85 , srv447668814:85 , srv569339610:85 , srv649508455:85 , srv692177885:85 , srv698002097:85 , srv741718104:85 , srv886431189:85 , srv1011265298:86 , srv1019806956:86 , srv1067872885:86 , srv1305030422:86 , srv1395555021:86 , srv1442388113:86 , srv1512366016:86 , srv1513519482:86 , srv1535632161:86 , srv1671217143:86 , srv2022548994:86 , srv295691195:86 , srv535664243:86 , srv968400860:86 , srv1064576425:87 , srv1090743316:87 , srv1432022010:87 , srv1456771847:87 , srv1493047310:87 , srv1524853683:87 , srv1566597924:87 , srv1742342609:87 , srv1797431387:87 , srv1832871775:87 , srv1958748404:87 , srv20377607:87 , srv2063601679:87 , srv2081401803:87 , srv2121120930:87 , srv2125727028:87 , srv255937429:87 , srv27380108:87 , srv31223744:87 , srv401232733:87 , srv458009397:87 , srv458409058:87 , srv516711512:87 , srv695788418:87 , srv735784904:87 , srv834081746:87 , srv845051879:87 , srv913163185:87 , srv939883570:87 , srv1023332809:88 , srv1236163417:88 , srv2109864750:88 , srv258801848:88 , srv526259885:88 , srv358894402:89 , srv66694292:89 , srv722530749:89 , srv712899409:532 } 2023-02-28 14:55:43,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:43,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272000377=60, srv877338351=93, srv526259885=74, srv1023332809=2, srv838772264=90, srv953963333=98, srv280752567=62, srv1532019594=33, srv1031145770=3, srv141197585=24, srv1412119170=25, srv1193993359=13, srv2006886002=49, srv1203150557=14, srv913163185=95, srv458409058=71, srv458009397=70, srv1512366016=30, srv1566597924=36, srv1671217143=38, srv2063601679=52, srv1513519482=31, srv213993090=57, srv258801848=59, srv1042623447=4, srv929746468=96, srv869686140=92, srv886431189=94, srv939883570=97, srv2022548994=50, srv1442388113=27, srv516711512=73, srv535664243=76, srv1742342609=40, srv401232733=67, srv1958748404=48, srv358894402=66, srv440819952=68, srv70228209=84, srv2121120930=55, srv1832871775=44, srv1926661269=47, srv1090743316=10, srv1794447563=42, srv735784904=87, srv1067872885=8, srv20377607=51, srv712899409=85, srv845051879=91, srv695788418=82, srv1395555021=22, srv1456771847=28, srv1367499866=19, srv1399951906=23, srv66694292=80, srv135480642=18, srv649508455=79, srv1188631627=12, srv27380108=61, srv1058668961=6, srv1236163417=16, srv447668814=69, srv476992859=72, srv1392936988=20, srv1535632161=34, srv2125727028=56, srv1088525795=9, srv1711766728=39, srv569339610=78, srv329870529=65, srv1305030422=17, srv553913771=77, srv139363433=21, srv1064576425=7, srv968400860=99, srv722530749=86, srv1432022010=26, srv1144880579=11, srv1857736799=45, srv1921653455=46, srv295691195=63, srv526571057=75, srv1493047310=29, srv1797431387=43, srv698002097=83, srv2081401803=53, srv1524853683=32, srv1044106307=5, srv741718104=88, srv1205202746=15, srv834081746=89, srv1011265298=0, srv255937429=58, srv1773282907=41, srv31223744=64, srv1610055821=37, srv2109864750=54, srv1019806956=1, srv1556302173=35, srv692177885=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-28 14:55:43,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-28 14:55:43,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-28 14:55:43,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-28 14:55:43,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-28 14:55:43,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-28 14:55:43,535 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008469335398956192); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.017102057161260623); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008469335398956192); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:43,535 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-28 14:55:43,535 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.20521707335799E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008469335398956192); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.017102057161260623); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008469335398956192); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-28 14:55:43,831 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 391 ms to try 5000 different iterations. Found a solution that moves 335 regions; Going from a computed imbalance of 9.20521707335799E-4 to a new imbalance of 6.933138469433061E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026801008406912323); MoveCostFunction : (multiplier=7.0, imbalance=0.03722222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:43,832 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv329870529:82 , srv401232733:82 , srv1305030422:83 , srv1610055821:83 , srv2022548994:83 , srv20377607:83 , srv458009397:83 , srv834081746:83 , srv1058668961:84 , srv139363433:84 , srv1399951906:84 , srv1535632161:84 , srv1566597924:84 , srv1773282907:84 , srv1832871775:84 , srv1857736799:84 , srv2063601679:84 , srv2081401803:84 , srv255937429:84 , srv258801848:84 , srv280752567:84 , srv476992859:84 , srv526571057:84 , srv553913771:84 , srv66694292:84 , srv692177885:84 , srv698002097:84 , srv722530749:84 , srv735784904:84 , srv838772264:84 , srv913163185:84 , srv939883570:84 , srv1023332809:85 , srv1031145770:85 , srv1042623447:85 , srv1044106307:85 , srv1064576425:85 , srv1067872885:85 , srv1088525795:85 , srv1193993359:85 , srv1203150557:85 , srv1205202746:85 , srv1236163417:85 , srv135480642:85 , srv1367499866:85 , srv1392936988:85 , srv1395555021:85 , srv1412119170:85 , srv1456771847:85 , srv1493047310:85 , srv1512366016:85 , srv1513519482:85 , srv1532019594:85 , srv1556302173:85 , srv1671217143:85 , srv1711766728:85 , srv1742342609:85 , srv1797431387:85 , srv1921653455:85 , srv1926661269:85 , srv2006886002:85 , srv2109864750:85 , srv2121120930:85 , srv2125727028:85 , srv213993090:85 , srv272000377:85 , srv27380108:85 , srv295691195:85 , srv31223744:85 , srv358894402:85 , srv447668814:85 , srv458409058:85 , srv516711512:85 , srv526259885:85 , srv535664243:85 , srv569339610:85 , srv649508455:85 , srv695788418:85 , srv741718104:85 , srv845051879:85 , srv929746468:85 , srv1011265298:86 , srv1019806956:86 , srv1090743316:86 , srv1188631627:86 , srv141197585:86 , srv1432022010:86 , srv1442388113:86 , srv1524853683:86 , srv1794447563:86 , srv1958748404:86 , srv440819952:86 , srv70228209:86 , srv869686140:86 , srv877338351:86 , srv886431189:86 , srv953963333:86 , srv968400860:86 , srv1144880579:87 , srv712899409:608 } 2023-02-28 14:55:43,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:43,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272000377=60, srv877338351=93, srv526259885=74, srv1023332809=2, srv838772264=90, srv953963333=98, srv280752567=62, srv1532019594=33, srv1031145770=3, srv141197585=24, srv1412119170=25, srv1193993359=13, srv2006886002=49, srv1203150557=14, srv913163185=95, srv458409058=71, srv458009397=70, srv1512366016=30, srv1566597924=36, srv1671217143=38, srv2063601679=52, srv1513519482=31, srv213993090=57, srv258801848=59, srv1042623447=4, srv929746468=96, srv869686140=92, srv886431189=94, srv939883570=97, srv2022548994=50, srv1442388113=27, srv516711512=73, srv535664243=76, srv1742342609=40, srv401232733=67, srv1958748404=48, srv358894402=66, srv440819952=68, srv70228209=84, srv2121120930=55, srv1832871775=44, srv1926661269=47, srv1090743316=10, srv1794447563=42, srv735784904=87, srv1067872885=8, srv20377607=51, srv712899409=85, srv845051879=91, srv695788418=82, srv1395555021=22, srv1456771847=28, srv1367499866=19, srv1399951906=23, srv66694292=80, srv135480642=18, srv649508455=79, srv1188631627=12, srv27380108=61, srv1058668961=6, srv1236163417=16, srv447668814=69, srv476992859=72, srv1392936988=20, srv1535632161=34, srv2125727028=56, srv1088525795=9, srv1711766728=39, srv569339610=78, srv329870529=65, srv1305030422=17, srv553913771=77, srv139363433=21, srv1064576425=7, srv968400860=99, srv722530749=86, srv1432022010=26, srv1144880579=11, srv1857736799=45, srv1921653455=46, srv295691195=63, srv526571057=75, srv1493047310=29, srv1797431387=43, srv698002097=83, srv2081401803=53, srv1524853683=32, srv1044106307=5, srv741718104=88, srv1205202746=15, srv834081746=89, srv1011265298=0, srv255937429=58, srv1773282907=41, srv31223744=64, srv1610055821=37, srv2109864750=54, srv1019806956=1, srv1556302173=35, srv692177885=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:43,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:43,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:43,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:43,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-28 14:55:43,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-28 14:55:43,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-28 14:55:43,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-28 14:55:43,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-28 14:55:43,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-28 14:55:43,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-28 14:55:43,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-28 14:55:43,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-28 14:55:43,962 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026801008406912323); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:43,962 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-28 14:55:43,963 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.698566455824883E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026801008406912323); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-28 14:55:44,158 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 284 ms to try 5000 different iterations. Found a solution that moves 58 regions; Going from a computed imbalance of 6.698566455824883E-5 to a new imbalance of 6.804110352546544E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0064444444444444445); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:44,159 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1090743316:83 , srv1144880579:83 , srv1203150557:83 , srv1236163417:83 , srv1432022010:83 , srv1442388113:83 , srv1711766728:83 , srv1794447563:83 , srv358894402:83 , srv66694292:83 , srv1011265298:84 , srv1019806956:84 , srv1023332809:84 , srv1031145770:84 , srv1042623447:84 , srv1044106307:84 , srv1058668961:84 , srv1064576425:84 , srv1067872885:84 , srv1088525795:84 , srv1188631627:84 , srv1193993359:84 , srv1205202746:84 , srv135480642:84 , srv1367499866:84 , srv1392936988:84 , srv139363433:84 , srv1395555021:84 , srv1399951906:84 , srv141197585:84 , srv1412119170:84 , srv1456771847:84 , srv1493047310:84 , srv1512366016:84 , srv1513519482:84 , srv1524853683:84 , srv1532019594:84 , srv1535632161:84 , srv1566597924:84 , srv1671217143:84 , srv1742342609:84 , srv1773282907:84 , srv1797431387:84 , srv1832871775:84 , srv1926661269:84 , srv1958748404:84 , srv2022548994:84 , srv20377607:84 , srv2063601679:84 , srv2109864750:84 , srv2121120930:84 , srv2125727028:84 , srv213993090:84 , srv255937429:84 , srv258801848:84 , srv272000377:84 , srv27380108:84 , srv280752567:84 , srv295691195:84 , srv31223744:84 , srv401232733:84 , srv440819952:84 , srv447668814:84 , srv458009397:84 , srv458409058:84 , srv476992859:84 , srv516711512:84 , srv526259885:84 , srv526571057:84 , srv535664243:84 , srv553913771:84 , srv569339610:84 , srv649508455:84 , srv692177885:84 , srv695788418:84 , srv698002097:84 , srv70228209:84 , srv722530749:84 , srv735784904:84 , srv741718104:84 , srv834081746:84 , srv838772264:84 , srv845051879:84 , srv869686140:84 , srv877338351:84 , srv886431189:84 , srv913163185:84 , srv929746468:84 , srv953963333:84 , srv968400860:84 , srv1305030422:85 , srv1556302173:85 , srv1610055821:85 , srv1857736799:85 , srv1921653455:85 , srv2006886002:85 , srv2081401803:85 , srv329870529:85 , srv939883570:85 , srv712899409:685 } 2023-02-28 14:55:44,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:44,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272000377=60, srv877338351=93, srv526259885=74, srv1023332809=2, srv838772264=90, srv953963333=98, srv280752567=62, srv1532019594=33, srv1031145770=3, srv141197585=24, srv1412119170=25, srv1193993359=13, srv2006886002=49, srv1203150557=14, srv913163185=95, srv458409058=71, srv458009397=70, srv1512366016=30, srv1566597924=36, srv1671217143=38, srv2063601679=52, srv1513519482=31, srv213993090=57, srv258801848=59, srv1042623447=4, srv929746468=96, srv869686140=92, srv886431189=94, srv939883570=97, srv2022548994=50, srv1442388113=27, srv516711512=73, srv535664243=76, srv1742342609=40, srv401232733=67, srv1958748404=48, srv358894402=66, srv440819952=68, srv70228209=84, srv2121120930=55, srv1832871775=44, srv1926661269=47, srv1090743316=10, srv1794447563=42, srv735784904=87, srv1067872885=8, srv20377607=51, srv712899409=85, srv845051879=91, srv695788418=82, srv1395555021=22, srv1456771847=28, srv1367499866=19, srv1399951906=23, srv66694292=80, srv135480642=18, srv649508455=79, srv1188631627=12, srv27380108=61, srv1058668961=6, srv1236163417=16, srv447668814=69, srv476992859=72, srv1392936988=20, srv1535632161=34, srv2125727028=56, srv1088525795=9, srv1711766728=39, srv569339610=78, srv329870529=65, srv1305030422=17, srv553913771=77, srv139363433=21, srv1064576425=7, srv968400860=99, srv722530749=86, srv1432022010=26, srv1144880579=11, srv1857736799=45, srv1921653455=46, srv295691195=63, srv526571057=75, srv1493047310=29, srv1797431387=43, srv698002097=83, srv2081401803=53, srv1524853683=32, srv1044106307=5, srv741718104=88, srv1205202746=15, srv834081746=89, srv1011265298=0, srv255937429=58, srv1773282907=41, srv31223744=64, srv1610055821=37, srv2109864750=54, srv1019806956=1, srv1556302173=35, srv692177885=81} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:44,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:44,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:44,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-28 14:55:44,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-28 14:55:44,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-28 14:55:44,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-28 14:55:44,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-28 14:55:44,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-28 14:55:44,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-28 14:55:44,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-28 14:55:44,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-28 14:55:44,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-28 14:55:44,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-28 14:55:44,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-28 14:55:44,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-28 14:55:44,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-28 14:55:44,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-28 14:55:44,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-28 14:55:44,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-28 14:55:44,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-28 14:55:44,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-28 14:55:44,303 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:44,304 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1090743316:83 , srv1144880579:83 , srv1203150557:83 , srv1236163417:83 , srv1432022010:83 , srv1442388113:83 , srv1711766728:83 , srv1794447563:83 , srv358894402:83 , srv66694292:83 , srv1011265298:84 , srv1019806956:84 , srv1023332809:84 , srv1031145770:84 , srv1042623447:84 , srv1044106307:84 , srv1058668961:84 , srv1064576425:84 , srv1067872885:84 , srv1088525795:84 , srv1188631627:84 , srv1193993359:84 , srv1205202746:84 , srv135480642:84 , srv1367499866:84 , srv1392936988:84 , srv139363433:84 , srv1395555021:84 , srv1399951906:84 , srv141197585:84 , srv1412119170:84 , srv1456771847:84 , srv1493047310:84 , srv1512366016:84 , srv1513519482:84 , srv1524853683:84 , srv1532019594:84 , srv1535632161:84 , srv1566597924:84 , srv1671217143:84 , srv1742342609:84 , srv1773282907:84 , srv1797431387:84 , srv1832871775:84 , srv1926661269:84 , srv1958748404:84 , srv2022548994:84 , srv20377607:84 , srv2063601679:84 , srv2109864750:84 , srv2121120930:84 , srv2125727028:84 , srv213993090:84 , srv255937429:84 , srv258801848:84 , srv272000377:84 , srv27380108:84 , srv280752567:84 , srv295691195:84 , srv31223744:84 , srv401232733:84 , srv440819952:84 , srv447668814:84 , srv458009397:84 , srv458409058:84 , srv476992859:84 , srv516711512:84 , srv526259885:84 , srv526571057:84 , srv535664243:84 , srv553913771:84 , srv569339610:84 , srv649508455:84 , srv692177885:84 , srv695788418:84 , srv698002097:84 , srv70228209:84 , srv722530749:84 , srv735784904:84 , srv741718104:84 , srv834081746:84 , srv838772264:84 , srv845051879:84 , srv869686140:84 , srv877338351:84 , srv886431189:84 , srv913163185:84 , srv929746468:84 , srv953963333:84 , srv968400860:84 , srv1305030422:85 , srv1556302173:85 , srv1610055821:85 , srv1857736799:85 , srv1921653455:85 , srv2006886002:85 , srv2081401803:85 , srv329870529:85 , srv939883570:85 , srv712899409:685 } 2023-02-28 14:55:44,335 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=267 (was 267), ProcessCount=165 (was 165), AvailableMemoryMB=4832 (was 5366) 2023-02-28 14:55:44,344 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=267, ProcessCount=165, AvailableMemoryMB=4832 2023-02-28 14:55:44,345 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-28 14:55:44,345 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-28 14:55:44,357 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012600287:84 , srv10490767:84 , srv1070641865:84 , srv1169009115:84 , srv1268781574:84 , srv1315734266:84 , srv1346773788:84 , srv1685318912:84 , srv1727396268:84 , srv1949532929:84 , srv1977632808:84 , srv2032141770:84 , srv2037990485:84 , srv2051882732:84 , srv2078522770:84 , srv2089407853:84 , srv2113988987:84 , srv2125416205:84 , srv217138976:84 , srv340920542:84 , srv38752468:84 , srv468916220:84 , srv529032436:84 , srv55266355:84 , srv571288150:84 , srv696828451:84 , srv751369717:84 , srv833553133:84 , srv99036644:84 , srv913722972:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-28 14:55:44,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:55:44,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv696828451=25, srv2078522770=14, srv1012600287=0, srv2032141770=11, srv1346773788=6, srv2113988987=16, srv1268781574=4, srv1315734266=5, srv2051882732=13, srv571288150=24, srv340920542=19, srv55266355=23, srv1949532929=9, srv10490767=1, srv1977632808=10, srv751369717=26, srv1727396268=8, srv1169009115=3, srv468916220=21, srv913722972=28, srv2037990485=12, srv99036644=29, srv38752468=20, srv217138976=18, srv2125416205=17, srv1070641865=2, srv1685318912=7, srv529032436=22, srv833553133=27, srv2089407853=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:55:44,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:55:44,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:55:44,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:55:44,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-28 14:55:44,398 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1594561972241093, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:55:44,398 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912695484535008, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1594561972241093, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-28 14:57:44,379 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9101886 different iterations. Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912695484535008 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:57:44,384 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1012600287:90 , srv10490767:90 , srv1070641865:90 , srv1169009115:90 , srv1268781574:90 , srv1315734266:90 , srv1346773788:90 , srv1685318912:90 , srv1727396268:90 , srv1949532929:90 , srv1977632808:90 , srv2032141770:90 , srv2037990485:90 , srv2051882732:90 , srv2078522770:90 , srv2089407853:90 , srv2113988987:90 , srv2125416205:90 , srv217138976:90 , srv340920542:90 , srv38752468:90 , srv468916220:90 , srv529032436:90 , srv55266355:90 , srv571288150:90 , srv696828451:90 , srv751369717:90 , srv833553133:90 , srv913722972:90 , srv99036644:90 } 2023-02-28 14:57:44,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-28 14:57:44,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv696828451=25, srv2078522770=14, srv1012600287=0, srv2032141770=11, srv1346773788=6, srv2113988987=16, srv1268781574=4, srv1315734266=5, srv2051882732=13, srv571288150=24, srv340920542=19, srv55266355=23, srv1949532929=9, srv10490767=1, srv1977632808=10, srv751369717=26, srv1727396268=8, srv1169009115=3, srv468916220=21, srv913722972=28, srv2037990485=12, srv99036644=29, srv38752468=20, srv217138976=18, srv2125416205=17, srv1070641865=2, srv1685318912=7, srv529032436=22, srv833553133=27, srv2089407853=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-28 14:57:44,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 14:57:44,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-28 14:57:44,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-28 14:57:44,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-28 14:57:44,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-28 14:57:44,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-28 14:57:44,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-28 14:57:44,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-28 14:57:44,399 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1012600287:90 , srv10490767:90 , srv1070641865:90 , srv1169009115:90 , srv1268781574:90 , srv1315734266:90 , srv1346773788:90 , srv1685318912:90 , srv1727396268:90 , srv1949532929:90 , srv1977632808:90 , srv2032141770:90 , srv2037990485:90 , srv2051882732:90 , srv2078522770:90 , srv2089407853:90 , srv2113988987:90 , srv2125416205:90 , srv217138976:90 , srv340920542:90 , srv38752468:90 , srv468916220:90 , srv529032436:90 , srv55266355:90 , srv571288150:90 , srv696828451:90 , srv751369717:90 , srv833553133:90 , srv913722972:90 , srv99036644:90 } 2023-02-28 14:57:44,409 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=186 (was 267), ProcessCount=164 (was 165), AvailableMemoryMB=5191 (was 4832) - AvailableMemoryMB LEAK? -