2023-11-06 15:55:36,873 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-11-06 15:55:37,144 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-11-06 15:55:37,300 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-11-06 15:55:37,378 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-06 15:55:37,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-06 15:55:37,404 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=184, ProcessCount=167, AvailableMemoryMB=15510 2023-11-06 15:55:37,407 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-06 15:55:37,407 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-06 15:55:37,563 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv105026448:84 , srv1059467075:84 , srv1080091429:84 , srv1176491111:84 , srv1183961738:84 , srv1186267294:84 , srv1211828987:84 , srv1231278297:84 , srv1276068865:84 , srv1279999649:84 , srv1347541318:84 , srv1351681639:84 , srv1359209164:84 , srv137567693:84 , srv1390648188:84 , srv1404184297:84 , srv1422185519:84 , srv1429360326:84 , srv1449449211:84 , srv1452279910:84 , srv14562167:84 , srv14788065:84 , srv1484157648:84 , srv1494342591:84 , srv1539102492:84 , srv1546045168:84 , srv1549688539:84 , srv1577975820:84 , srv1585384242:84 , srv1586797640:84 , srv1619688394:84 , srv1629245589:84 , srv1642892145:84 , srv1646854670:84 , srv1708289188:84 , srv1721343851:84 , srv1819864177:84 , srv1845351521:84 , srv1866677903:84 , srv1885117516:84 , srv1886189810:84 , srv1934073784:84 , srv1961618918:84 , srv198377379:84 , srv1991753705:84 , srv2037834008:84 , srv2046204452:84 , srv2053680444:84 , srv2056081221:84 , srv2060094291:84 , srv2091548074:84 , srv2127316930:84 , srv2130810709:84 , srv219627421:84 , srv229476427:84 , srv262117374:84 , srv281659069:84 , srv321991214:84 , srv34919596:84 , srv352895074:84 , srv355537689:84 , srv367957203:84 , srv370099941:84 , srv370586135:84 , srv407249794:84 , srv427448689:84 , srv465026042:84 , srv495489521:84 , srv529759842:84 , srv54773514:84 , srv577669071:84 , srv590482886:84 , srv592013109:84 , srv598935145:84 , srv607741054:84 , srv636764983:84 , srv675409308:84 , srv683081551:84 , srv694040944:84 , srv741363271:84 , srv764125955:84 , srv767216661:84 , srv77298871:84 , srv79025825:84 , srv806026123:84 , srv817224083:84 , srv82469433:84 , srv840608642:84 , srv849344618:84 , srv862887695:84 , srv863127827:84 , srv864682039:84 , srv891963239:84 , srv893963800:84 , srv911489656:84 , srv95041980:84 , srv950672157:84 , srv959933663:84 , srv976082808:84 , srv704103410:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-11-06 15:55:37,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:37,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991753705=44, srv1059467075=1, srv495489521=67, srv1276068865=8, srv465026042=66, srv1886189810=40, srv1619688394=30, srv262117374=55, srv840608642=88, srv14788065=21, srv2060094291=49, srv1359209164=12, srv1351681639=11, srv950672157=97, srv77298871=83, srv767216661=82, srv1422185519=16, srv105026448=0, srv741363271=80, srv1629245589=31, srv529759842=68, srv607741054=74, srv427448689=65, srv1347541318=10, srv367957203=61, srv2046204452=46, srv1845351521=37, srv1404184297=15, srv34919596=58, srv1585384242=28, srv1183961738=4, srv864682039=92, srv1176491111=3, srv817224083=86, srv1211828987=6, srv1080091429=2, srv577669071=70, srv976082808=99, srv1452279910=19, srv198377379=43, srv229476427=54, srv1429360326=17, srv14562167=20, srv598935145=73, srv54773514=69, srv1646854670=33, srv1546045168=25, srv1866677903=38, srv1885117516=39, srv683081551=77, srv407249794=64, srv849344618=89, srv1642892145=32, srv2053680444=47, srv1494342591=23, srv370099941=62, srv959933663=98, srv704103410=79, srv1186267294=5, srv2037834008=45, srv1449449211=18, srv95041980=96, srv321991214=57, srv911489656=95, srv636764983=75, srv592013109=72, srv2091548074=50, srv82469433=87, srv2056081221=48, srv1577975820=27, srv2127316930=51, srv764125955=81, srv1539102492=24, srv352895074=59, srv893963800=94, srv1934073784=41, srv862887695=90, srv137567693=13, srv355537689=60, srv1231278297=7, srv1819864177=36, srv1279999649=9, srv1961618918=42, srv1721343851=35, srv1586797640=29, srv1390648188=14, srv891963239=93, srv590482886=71, srv79025825=84, srv1484157648=22, srv219627421=53, srv806026123=85, srv863127827=91, srv2130810709=52, srv675409308=76, srv1549688539=26, srv281659069=56, srv694040944=78, srv1708289188=34, srv370586135=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:37,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:37,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:37,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-06 15:55:37,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-06 15:55:37,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-06 15:55:37,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-06 15:55:37,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-06 15:55:37,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-06 15:55:37,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-06 15:55:37,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-06 15:55:37,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-06 15:55:37,968 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:37,968 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-06 15:55:37,969 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-06 15:55:38,496 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 804 ms to try 5000 different iterations. Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18077755870199655. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040848953317747384, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05945695138890323, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040848953317747384, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:38,510 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1080091429:80 , srv1539102492:82 , srv34919596:82 , srv862887695:82 , srv893963800:82 , srv105026448:83 , srv1059467075:83 , srv1347541318:83 , srv1390648188:83 , srv1484157648:83 , srv1934073784:83 , srv198377379:83 , srv2037834008:83 , srv2053680444:83 , srv407249794:83 , srv694040944:83 , srv976082808:83 , srv1231278297:84 , srv1351681639:84 , srv1845351521:84 , srv219627421:84 , srv465026042:84 , srv767216661:84 , srv864682039:84 , srv959933663:84 , srv1449449211:85 , srv1549688539:85 , srv1586797640:85 , srv1629245589:85 , srv1819864177:85 , srv1886189810:85 , srv2130810709:85 , srv355537689:85 , srv370586135:85 , srv427448689:85 , srv495489521:85 , srv675409308:85 , srv840608642:85 , srv1183961738:86 , srv137567693:86 , srv1422185519:86 , srv14788065:86 , srv1494342591:86 , srv1546045168:86 , srv1646854670:86 , srv1708289188:86 , srv1866677903:86 , srv54773514:86 , srv590482886:86 , srv592013109:86 , srv607741054:86 , srv636764983:86 , srv741363271:86 , srv806026123:86 , srv849344618:86 , srv863127827:86 , srv1404184297:87 , srv1619688394:87 , srv1991753705:87 , srv2046204452:87 , srv281659069:87 , srv321991214:87 , srv367957203:87 , srv577669071:87 , srv683081551:87 , srv764125955:87 , srv817224083:87 , srv950672157:87 , srv1429360326:88 , srv1585384242:88 , srv1721343851:88 , srv1961618918:88 , srv2056081221:88 , srv2091548074:88 , srv2127316930:88 , srv229476427:88 , srv262117374:88 , srv598935145:88 , srv1186267294:89 , srv1211828987:89 , srv1642892145:89 , srv352895074:89 , srv77298871:89 , srv79025825:89 , srv911489656:89 , srv95041980:89 , srv1176491111:90 , srv1359209164:90 , srv14562167:90 , srv1885117516:90 , srv2060094291:90 , srv529759842:90 , srv82469433:90 , srv1279999649:91 , srv1577975820:91 , srv370099941:91 , srv891963239:91 , srv1276068865:93 , srv1452279910:95 , srv704103410:453 } 2023-11-06 15:55:38,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:38,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991753705=44, srv1059467075=1, srv495489521=67, srv1276068865=8, srv465026042=66, srv1886189810=40, srv1619688394=30, srv262117374=55, srv840608642=88, srv14788065=21, srv2060094291=49, srv1359209164=12, srv1351681639=11, srv950672157=97, srv77298871=83, srv767216661=82, srv1422185519=16, srv105026448=0, srv741363271=80, srv1629245589=31, srv529759842=68, srv607741054=74, srv427448689=65, srv1347541318=10, srv367957203=61, srv2046204452=46, srv1845351521=37, srv1404184297=15, srv34919596=58, srv1585384242=28, srv1183961738=4, srv864682039=92, srv1176491111=3, srv817224083=86, srv1211828987=6, srv1080091429=2, srv577669071=70, srv976082808=99, srv1452279910=19, srv198377379=43, srv229476427=54, srv1429360326=17, srv14562167=20, srv598935145=73, srv54773514=69, srv1646854670=33, srv1546045168=25, srv1866677903=38, srv1885117516=39, srv683081551=77, srv407249794=64, srv849344618=89, srv1642892145=32, srv2053680444=47, srv1494342591=23, srv370099941=62, srv959933663=98, srv704103410=79, srv1186267294=5, srv2037834008=45, srv1449449211=18, srv95041980=96, srv321991214=57, srv911489656=95, srv636764983=75, srv592013109=72, srv2091548074=50, srv82469433=87, srv2056081221=48, srv1577975820=27, srv2127316930=51, srv764125955=81, srv1539102492=24, srv352895074=59, srv893963800=94, srv1934073784=41, srv862887695=90, srv137567693=13, srv355537689=60, srv1231278297=7, srv1819864177=36, srv1279999649=9, srv1961618918=42, srv1721343851=35, srv1586797640=29, srv1390648188=14, srv891963239=93, srv590482886=71, srv79025825=84, srv1484157648=22, srv219627421=53, srv806026123=85, srv863127827=91, srv2130810709=52, srv675409308=76, srv1549688539=26, srv281659069=56, srv694040944=78, srv1708289188=34, srv370586135=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-06 15:55:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-06 15:55:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-06 15:55:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-06 15:55:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-06 15:55:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-06 15:55:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-06 15:55:38,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-06 15:55:38,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-06 15:55:38,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-06 15:55:38,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-06 15:55:38,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-06 15:55:38,657 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040848953317747384, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05945695138890323, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040848953317747384, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:38,657 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-06 15:55:38,657 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1807447326243907, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040848953317747384, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05945695138890323, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040848953317747384, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-06 15:55:38,861 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations. Found a solution that moves 2581 regions; Going from a computed imbalance of 0.1807447326243907 to a new imbalance of 0.016118801984714548. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022492187334354038); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037774806775627945, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2867777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022492187334354038); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:38,872 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1452279910:78 , srv370099941:79 , srv1276068865:80 , srv891963239:80 , srv1279999649:81 , srv2127316930:81 , srv911489656:81 , srv1211828987:82 , srv14562167:82 , srv1577975820:82 , srv1885117516:82 , srv2060094291:82 , srv529759842:82 , srv82469433:82 , srv95041980:82 , srv1176491111:83 , srv1359209164:83 , srv1642892145:83 , srv2091548074:83 , srv262117374:83 , srv352895074:83 , srv79025825:83 , srv1186267294:84 , srv1429360326:84 , srv1549688539:84 , srv1619688394:84 , srv1961618918:84 , srv229476427:84 , srv427448689:84 , srv636764983:84 , srv683081551:84 , srv741363271:84 , srv806026123:84 , srv1404184297:85 , srv1585384242:85 , srv1991753705:85 , srv2046204452:85 , srv2056081221:85 , srv281659069:85 , srv367957203:85 , srv598935145:85 , srv675409308:85 , srv764125955:85 , srv77298871:85 , srv137567693:86 , srv1422185519:86 , srv1449449211:86 , srv14788065:86 , srv1494342591:86 , srv1708289188:86 , srv1866677903:86 , srv321991214:86 , srv355537689:86 , srv590482886:86 , srv607741054:86 , srv863127827:86 , srv950672157:86 , srv1059467075:87 , srv1183961738:87 , srv1347541318:87 , srv1546045168:87 , srv1646854670:87 , srv1721343851:87 , srv1845351521:87 , srv1886189810:87 , srv1934073784:87 , srv370586135:87 , srv495489521:87 , srv592013109:87 , srv817224083:87 , srv840608642:87 , srv849344618:87 , srv959933663:87 , srv1231278297:88 , srv1390648188:88 , srv1484157648:88 , srv1586797640:88 , srv1629245589:88 , srv1819864177:88 , srv2130810709:88 , srv219627421:88 , srv407249794:88 , srv767216661:88 , srv105026448:89 , srv1080091429:89 , srv1351681639:89 , srv1539102492:89 , srv198377379:89 , srv54773514:89 , srv577669071:89 , srv694040944:89 , srv864682039:89 , srv2037834008:90 , srv2053680444:90 , srv34919596:90 , srv862887695:90 , srv976082808:90 , srv465026042:91 , srv893963800:91 , srv704103410:521 } 2023-11-06 15:55:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991753705=44, srv1059467075=1, srv495489521=67, srv1276068865=8, srv465026042=66, srv1886189810=40, srv1619688394=30, srv262117374=55, srv840608642=88, srv14788065=21, srv2060094291=49, srv1359209164=12, srv1351681639=11, srv950672157=97, srv77298871=83, srv767216661=82, srv1422185519=16, srv105026448=0, srv741363271=80, srv1629245589=31, srv529759842=68, srv607741054=74, srv427448689=65, srv1347541318=10, srv367957203=61, srv2046204452=46, srv1845351521=37, srv1404184297=15, srv34919596=58, srv1585384242=28, srv1183961738=4, srv864682039=92, srv1176491111=3, srv817224083=86, srv1211828987=6, srv1080091429=2, srv577669071=70, srv976082808=99, srv1452279910=19, srv198377379=43, srv229476427=54, srv1429360326=17, srv14562167=20, srv598935145=73, srv54773514=69, srv1646854670=33, srv1546045168=25, srv1866677903=38, srv1885117516=39, srv683081551=77, srv407249794=64, srv849344618=89, srv1642892145=32, srv2053680444=47, srv1494342591=23, srv370099941=62, srv959933663=98, srv704103410=79, srv1186267294=5, srv2037834008=45, srv1449449211=18, srv95041980=96, srv321991214=57, srv911489656=95, srv636764983=75, srv592013109=72, srv2091548074=50, srv82469433=87, srv2056081221=48, srv1577975820=27, srv2127316930=51, srv764125955=81, srv1539102492=24, srv352895074=59, srv893963800=94, srv1934073784=41, srv862887695=90, srv137567693=13, srv355537689=60, srv1231278297=7, srv1819864177=36, srv1279999649=9, srv1961618918=42, srv1721343851=35, srv1586797640=29, srv1390648188=14, srv891963239=93, srv590482886=71, srv79025825=84, srv1484157648=22, srv219627421=53, srv806026123=85, srv863127827=91, srv2130810709=52, srv675409308=76, srv1549688539=26, srv281659069=56, srv694040944=78, srv1708289188=34, srv370586135=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-06 15:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-06 15:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-06 15:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-06 15:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-06 15:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-06 15:55:38,990 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022492187334354038); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037774806775627945, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022492187334354038); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:38,990 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-06 15:55:38,991 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01610072943644223, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022492187334354038); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037774806775627945, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022492187334354038); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-06 15:55:39,150 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 239 ms to try 5000 different iterations. Found a solution that moves 969 regions; Going from a computed imbalance of 0.01610072943644223 to a new imbalance of 0.00130251916426516. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007680163464195452); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011301400844806558); MoveCostFunction : (multiplier=7.0, imbalance=0.10766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007680163464195452); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,152 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1721343851:82 , srv1176491111:83 , srv2130810709:83 , srv577669071:83 , srv77298871:83 , srv817224083:83 , srv976082808:83 , srv1183961738:84 , srv1452279910:84 , srv1494342591:84 , srv1546045168:84 , srv1577975820:84 , srv1629245589:84 , srv1646854670:84 , srv2053680444:84 , srv2056081221:84 , srv229476427:84 , srv321991214:84 , srv465026042:84 , srv54773514:84 , srv598935145:84 , srv82469433:84 , srv840608642:84 , srv849344618:84 , srv864682039:84 , srv893963800:84 , srv950672157:84 , srv105026448:85 , srv1186267294:85 , srv1231278297:85 , srv1276068865:85 , srv1351681639:85 , srv1359209164:85 , srv137567693:85 , srv1404184297:85 , srv1422185519:85 , srv1429360326:85 , srv14788065:85 , srv1585384242:85 , srv1586797640:85 , srv1642892145:85 , srv1708289188:85 , srv1819864177:85 , srv1845351521:85 , srv1886189810:85 , srv1961618918:85 , srv198377379:85 , srv2037834008:85 , srv219627421:85 , srv281659069:85 , srv34919596:85 , srv370586135:85 , srv590482886:85 , srv592013109:85 , srv607741054:85 , srv694040944:85 , srv862887695:85 , srv863127827:85 , srv1211828987:86 , srv1279999649:86 , srv1390648188:86 , srv1449449211:86 , srv14562167:86 , srv1484157648:86 , srv1539102492:86 , srv1619688394:86 , srv1885117516:86 , srv1991753705:86 , srv2046204452:86 , srv2091548074:86 , srv262117374:86 , srv352895074:86 , srv367957203:86 , srv407249794:86 , srv495489521:86 , srv529759842:86 , srv675409308:86 , srv683081551:86 , srv764125955:86 , srv767216661:86 , srv79025825:86 , srv806026123:86 , srv891963239:86 , srv959933663:86 , srv1059467075:87 , srv1866677903:87 , srv2060094291:87 , srv355537689:87 , srv370099941:87 , srv911489656:87 , srv1080091429:88 , srv1347541318:88 , srv1934073784:88 , srv2127316930:88 , srv427448689:88 , srv636764983:88 , srv741363271:88 , srv95041980:89 , srv1549688539:90 , srv704103410:552 } 2023-11-06 15:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:39,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991753705=44, srv1059467075=1, srv495489521=67, srv1276068865=8, srv465026042=66, srv1886189810=40, srv1619688394=30, srv262117374=55, srv840608642=88, srv14788065=21, srv2060094291=49, srv1359209164=12, srv1351681639=11, srv950672157=97, srv77298871=83, srv767216661=82, srv1422185519=16, srv105026448=0, srv741363271=80, srv1629245589=31, srv529759842=68, srv607741054=74, srv427448689=65, srv1347541318=10, srv367957203=61, srv2046204452=46, srv1845351521=37, srv1404184297=15, srv34919596=58, srv1585384242=28, srv1183961738=4, srv864682039=92, srv1176491111=3, srv817224083=86, srv1211828987=6, srv1080091429=2, srv577669071=70, srv976082808=99, srv1452279910=19, srv198377379=43, srv229476427=54, srv1429360326=17, srv14562167=20, srv598935145=73, srv54773514=69, srv1646854670=33, srv1546045168=25, srv1866677903=38, srv1885117516=39, srv683081551=77, srv407249794=64, srv849344618=89, srv1642892145=32, srv2053680444=47, srv1494342591=23, srv370099941=62, srv959933663=98, srv704103410=79, srv1186267294=5, srv2037834008=45, srv1449449211=18, srv95041980=96, srv321991214=57, srv911489656=95, srv636764983=75, srv592013109=72, srv2091548074=50, srv82469433=87, srv2056081221=48, srv1577975820=27, srv2127316930=51, srv764125955=81, srv1539102492=24, srv352895074=59, srv893963800=94, srv1934073784=41, srv862887695=90, srv137567693=13, srv355537689=60, srv1231278297=7, srv1819864177=36, srv1279999649=9, srv1961618918=42, srv1721343851=35, srv1586797640=29, srv1390648188=14, srv891963239=93, srv590482886=71, srv79025825=84, srv1484157648=22, srv219627421=53, srv806026123=85, srv863127827=91, srv2130810709=52, srv675409308=76, srv1549688539=26, srv281659069=56, srv694040944=78, srv1708289188=34, srv370586135=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:39,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:39,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-06 15:55:39,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-06 15:55:39,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-06 15:55:39,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-06 15:55:39,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-06 15:55:39,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-06 15:55:39,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007680163464195452); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011301400844806558); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007680163464195452); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,262 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-06 15:55:39,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0012957340812446727, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007680163464195452); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011301400844806558); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007680163464195452); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-06 15:55:39,503 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 325 ms to try 5000 different iterations. Found a solution that moves 348 regions; Going from a computed imbalance of 0.0012957340812446727 to a new imbalance of 7.439811228904754E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.03866666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,504 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv95041980:82 , srv1549688539:83 , srv1866677903:83 , srv2060094291:83 , srv465026042:83 , srv54773514:83 , srv577669071:83 , srv767216661:83 , srv1347541318:84 , srv1351681639:84 , srv1359209164:84 , srv1404184297:84 , srv1452279910:84 , srv14562167:84 , srv1585384242:84 , srv1646854670:84 , srv1819864177:84 , srv1885117516:84 , srv1991753705:84 , srv2037834008:84 , srv2053680444:84 , srv262117374:84 , srv352895074:84 , srv355537689:84 , srv367957203:84 , srv370586135:84 , srv495489521:84 , srv529759842:84 , srv592013109:84 , srv636764983:84 , srv741363271:84 , srv764125955:84 , srv79025825:84 , srv911489656:84 , srv105026448:85 , srv1059467075:85 , srv1080091429:85 , srv1176491111:85 , srv1183961738:85 , srv1186267294:85 , srv1211828987:85 , srv1231278297:85 , srv1276068865:85 , srv1279999649:85 , srv137567693:85 , srv1390648188:85 , srv1422185519:85 , srv1449449211:85 , srv14788065:85 , srv1484157648:85 , srv1494342591:85 , srv1539102492:85 , srv1546045168:85 , srv1577975820:85 , srv1586797640:85 , srv1619688394:85 , srv1629245589:85 , srv1721343851:85 , srv1886189810:85 , srv1934073784:85 , srv1961618918:85 , srv198377379:85 , srv2046204452:85 , srv2056081221:85 , srv2091548074:85 , srv2127316930:85 , srv219627421:85 , srv281659069:85 , srv321991214:85 , srv34919596:85 , srv370099941:85 , srv407249794:85 , srv427448689:85 , srv590482886:85 , srv607741054:85 , srv683081551:85 , srv694040944:85 , srv77298871:85 , srv817224083:85 , srv849344618:85 , srv862887695:85 , srv863127827:85 , srv864682039:85 , srv891963239:85 , srv893963800:85 , srv950672157:85 , srv959933663:85 , srv1429360326:86 , srv1642892145:86 , srv1708289188:86 , srv1845351521:86 , srv2130810709:86 , srv229476427:86 , srv598935145:86 , srv675409308:86 , srv806026123:86 , srv82469433:86 , srv840608642:86 , srv976082808:86 , srv704103410:616 } 2023-11-06 15:55:39,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991753705=44, srv1059467075=1, srv495489521=67, srv1276068865=8, srv465026042=66, srv1886189810=40, srv1619688394=30, srv262117374=55, srv840608642=88, srv14788065=21, srv2060094291=49, srv1359209164=12, srv1351681639=11, srv950672157=97, srv77298871=83, srv767216661=82, srv1422185519=16, srv105026448=0, srv741363271=80, srv1629245589=31, srv529759842=68, srv607741054=74, srv427448689=65, srv1347541318=10, srv367957203=61, srv2046204452=46, srv1845351521=37, srv1404184297=15, srv34919596=58, srv1585384242=28, srv1183961738=4, srv864682039=92, srv1176491111=3, srv817224083=86, srv1211828987=6, srv1080091429=2, srv577669071=70, srv976082808=99, srv1452279910=19, srv198377379=43, srv229476427=54, srv1429360326=17, srv14562167=20, srv598935145=73, srv54773514=69, srv1646854670=33, srv1546045168=25, srv1866677903=38, srv1885117516=39, srv683081551=77, srv407249794=64, srv849344618=89, srv1642892145=32, srv2053680444=47, srv1494342591=23, srv370099941=62, srv959933663=98, srv704103410=79, srv1186267294=5, srv2037834008=45, srv1449449211=18, srv95041980=96, srv321991214=57, srv911489656=95, srv636764983=75, srv592013109=72, srv2091548074=50, srv82469433=87, srv2056081221=48, srv1577975820=27, srv2127316930=51, srv764125955=81, srv1539102492=24, srv352895074=59, srv893963800=94, srv1934073784=41, srv862887695=90, srv137567693=13, srv355537689=60, srv1231278297=7, srv1819864177=36, srv1279999649=9, srv1961618918=42, srv1721343851=35, srv1586797640=29, srv1390648188=14, srv891963239=93, srv590482886=71, srv79025825=84, srv1484157648=22, srv219627421=53, srv806026123=85, srv863127827=91, srv2130810709=52, srv675409308=76, srv1549688539=26, srv281659069=56, srv694040944=78, srv1708289188=34, srv370586135=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-06 15:55:39,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-06 15:55:39,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-06 15:55:39,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-06 15:55:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-06 15:55:39,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-06 15:55:39,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-06 15:55:39,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-06 15:55:39,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-06 15:55:39,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-06 15:55:39,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-06 15:55:39,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-06 15:55:39,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-06 15:55:39,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-06 15:55:39,620 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,620 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-06 15:55:39,621 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.196136420738648E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-06 15:55:39,777 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 242 ms to try 5000 different iterations. Found a solution that moves 50 regions; Going from a computed imbalance of 7.196136420738648E-5 to a new imbalance of 7.094176536829228E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.005555555555555556); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,778 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1186267294:83 , srv1276068865:83 , srv1279999649:83 , srv1429360326:83 , srv1586797640:83 , srv1642892145:83 , srv1708289188:83 , srv1934073784:83 , srv2046204452:83 , srv598935145:83 , srv105026448:84 , srv1059467075:84 , srv1080091429:84 , srv1183961738:84 , srv1211828987:84 , srv1231278297:84 , srv1347541318:84 , srv1351681639:84 , srv1359209164:84 , srv137567693:84 , srv1390648188:84 , srv1422185519:84 , srv1449449211:84 , srv14562167:84 , srv14788065:84 , srv1484157648:84 , srv1539102492:84 , srv1546045168:84 , srv1549688539:84 , srv1577975820:84 , srv1585384242:84 , srv1619688394:84 , srv1629245589:84 , srv1721343851:84 , srv1819864177:84 , srv1845351521:84 , srv1866677903:84 , srv1885117516:84 , srv1886189810:84 , srv1961618918:84 , srv198377379:84 , srv1991753705:84 , srv2037834008:84 , srv2056081221:84 , srv2060094291:84 , srv2091548074:84 , srv2127316930:84 , srv2130810709:84 , srv219627421:84 , srv229476427:84 , srv281659069:84 , srv321991214:84 , srv34919596:84 , srv352895074:84 , srv355537689:84 , srv367957203:84 , srv370099941:84 , srv407249794:84 , srv427448689:84 , srv465026042:84 , srv495489521:84 , srv529759842:84 , srv54773514:84 , srv577669071:84 , srv590482886:84 , srv592013109:84 , srv607741054:84 , srv636764983:84 , srv675409308:84 , srv683081551:84 , srv694040944:84 , srv741363271:84 , srv764125955:84 , srv77298871:84 , srv79025825:84 , srv806026123:84 , srv817224083:84 , srv82469433:84 , srv840608642:84 , srv849344618:84 , srv862887695:84 , srv863127827:84 , srv864682039:84 , srv891963239:84 , srv893963800:84 , srv95041980:84 , srv950672157:84 , srv959933663:84 , srv976082808:84 , srv1176491111:85 , srv1404184297:85 , srv1452279910:85 , srv1494342591:85 , srv1646854670:85 , srv2053680444:85 , srv262117374:85 , srv370586135:85 , srv767216661:85 , srv911489656:85 , srv704103410:684 } 2023-11-06 15:55:39,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:39,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991753705=44, srv1059467075=1, srv495489521=67, srv1276068865=8, srv465026042=66, srv1886189810=40, srv1619688394=30, srv262117374=55, srv840608642=88, srv14788065=21, srv2060094291=49, srv1359209164=12, srv1351681639=11, srv950672157=97, srv77298871=83, srv767216661=82, srv1422185519=16, srv105026448=0, srv741363271=80, srv1629245589=31, srv529759842=68, srv607741054=74, srv427448689=65, srv1347541318=10, srv367957203=61, srv2046204452=46, srv1845351521=37, srv1404184297=15, srv34919596=58, srv1585384242=28, srv1183961738=4, srv864682039=92, srv1176491111=3, srv817224083=86, srv1211828987=6, srv1080091429=2, srv577669071=70, srv976082808=99, srv1452279910=19, srv198377379=43, srv229476427=54, srv1429360326=17, srv14562167=20, srv598935145=73, srv54773514=69, srv1646854670=33, srv1546045168=25, srv1866677903=38, srv1885117516=39, srv683081551=77, srv407249794=64, srv849344618=89, srv1642892145=32, srv2053680444=47, srv1494342591=23, srv370099941=62, srv959933663=98, srv704103410=79, srv1186267294=5, srv2037834008=45, srv1449449211=18, srv95041980=96, srv321991214=57, srv911489656=95, srv636764983=75, srv592013109=72, srv2091548074=50, srv82469433=87, srv2056081221=48, srv1577975820=27, srv2127316930=51, srv764125955=81, srv1539102492=24, srv352895074=59, srv893963800=94, srv1934073784=41, srv862887695=90, srv137567693=13, srv355537689=60, srv1231278297=7, srv1819864177=36, srv1279999649=9, srv1961618918=42, srv1721343851=35, srv1586797640=29, srv1390648188=14, srv891963239=93, srv590482886=71, srv79025825=84, srv1484157648=22, srv219627421=53, srv806026123=85, srv863127827=91, srv2130810709=52, srv675409308=76, srv1549688539=26, srv281659069=56, srv694040944=78, srv1708289188=34, srv370586135=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:39,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:39,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:39,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:39,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-06 15:55:39,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-06 15:55:39,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-06 15:55:39,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-06 15:55:39,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-06 15:55:39,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-06 15:55:39,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-06 15:55:39,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-06 15:55:39,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-06 15:55:39,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-06 15:55:39,911 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,911 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1186267294:83 , srv1276068865:83 , srv1279999649:83 , srv1429360326:83 , srv1586797640:83 , srv1642892145:83 , srv1708289188:83 , srv1934073784:83 , srv2046204452:83 , srv598935145:83 , srv105026448:84 , srv1059467075:84 , srv1080091429:84 , srv1183961738:84 , srv1211828987:84 , srv1231278297:84 , srv1347541318:84 , srv1351681639:84 , srv1359209164:84 , srv137567693:84 , srv1390648188:84 , srv1422185519:84 , srv1449449211:84 , srv14562167:84 , srv14788065:84 , srv1484157648:84 , srv1539102492:84 , srv1546045168:84 , srv1549688539:84 , srv1577975820:84 , srv1585384242:84 , srv1619688394:84 , srv1629245589:84 , srv1721343851:84 , srv1819864177:84 , srv1845351521:84 , srv1866677903:84 , srv1885117516:84 , srv1886189810:84 , srv1961618918:84 , srv198377379:84 , srv1991753705:84 , srv2037834008:84 , srv2056081221:84 , srv2060094291:84 , srv2091548074:84 , srv2127316930:84 , srv2130810709:84 , srv219627421:84 , srv229476427:84 , srv281659069:84 , srv321991214:84 , srv34919596:84 , srv352895074:84 , srv355537689:84 , srv367957203:84 , srv370099941:84 , srv407249794:84 , srv427448689:84 , srv465026042:84 , srv495489521:84 , srv529759842:84 , srv54773514:84 , srv577669071:84 , srv590482886:84 , srv592013109:84 , srv607741054:84 , srv636764983:84 , srv675409308:84 , srv683081551:84 , srv694040944:84 , srv741363271:84 , srv764125955:84 , srv77298871:84 , srv79025825:84 , srv806026123:84 , srv817224083:84 , srv82469433:84 , srv840608642:84 , srv849344618:84 , srv862887695:84 , srv863127827:84 , srv864682039:84 , srv891963239:84 , srv893963800:84 , srv95041980:84 , srv950672157:84 , srv959933663:84 , srv976082808:84 , srv1176491111:85 , srv1404184297:85 , srv1452279910:85 , srv1494342591:85 , srv1646854670:85 , srv2053680444:85 , srv262117374:85 , srv370586135:85 , srv767216661:85 , srv911489656:85 , srv704103410:684 } 2023-11-06 15:55:39,930 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 184) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=15003 (was 15510) 2023-11-06 15:55:39,937 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=200, ProcessCount=167, AvailableMemoryMB=15003 2023-11-06 15:55:39,937 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-06 15:55:39,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-11-06 15:55:39,948 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1194705503:84 , srv1299310774:84 , srv1363486183:84 , srv1468222236:84 , srv1621590261:84 , srv1661627138:84 , srv1743782673:84 , srv1761126025:84 , srv1839257084:84 , srv1969155229:84 , srv1973712768:84 , srv197745920:84 , srv2024915786:84 , srv2071963710:84 , srv2127818193:84 , srv2146319371:84 , srv218440738:84 , srv348300747:84 , srv374311070:84 , srv38281468:84 , srv415866164:84 , srv480185491:84 , srv489756226:84 , srv605596685:84 , srv708514584:84 , srv762447764:84 , srv797661327:84 , srv877766955:84 , srv990184919:84 , srv409094444:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-11-06 15:55:39,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:55:39,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1839257084=8, srv708514584=25, srv1363486183=2, srv1973712768=10, srv374311070=18, srv990184919=29, srv1761126025=7, srv1743782673=6, srv1468222236=3, srv1661627138=5, srv1194705503=0, srv762447764=26, srv2127818193=14, srv797661327=27, srv1621590261=4, srv409094444=20, srv218440738=16, srv480185491=22, srv2024915786=12, srv415866164=21, srv38281468=19, srv197745920=11, srv489756226=23, srv877766955=28, srv2071963710=13, srv348300747=17, srv1969155229=9, srv605596685=24, srv1299310774=1, srv2146319371=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:55:39,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:55:39,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:55:39,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:55:39,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:55:39,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:55:39,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:55:39,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:55:39,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:55:39,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-06 15:55:39,988 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0819858858284963, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:55:39,988 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991245137811344, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0819858858284963, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-11-06 15:57:39,968 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9871595 different iterations. Found a solution that moves 1832 regions; Going from a computed imbalance of 0.991245137811344 to a new imbalance of 4.275979392340115E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6785185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:57:39,974 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1194705503:90 , srv1299310774:90 , srv1363486183:90 , srv1468222236:90 , srv1621590261:90 , srv1661627138:90 , srv1743782673:90 , srv1761126025:90 , srv1839257084:90 , srv1969155229:90 , srv1973712768:90 , srv197745920:90 , srv2024915786:90 , srv2071963710:90 , srv2127818193:90 , srv2146319371:90 , srv218440738:90 , srv348300747:90 , srv374311070:90 , srv38281468:90 , srv409094444:90 , srv415866164:90 , srv480185491:90 , srv489756226:90 , srv605596685:90 , srv708514584:90 , srv762447764:90 , srv797661327:90 , srv877766955:90 , srv990184919:90 } 2023-11-06 15:57:39,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-06 15:57:39,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1839257084=8, srv708514584=25, srv1363486183=2, srv1973712768=10, srv374311070=18, srv990184919=29, srv1761126025=7, srv1743782673=6, srv1468222236=3, srv1661627138=5, srv1194705503=0, srv762447764=26, srv2127818193=14, srv797661327=27, srv1621590261=4, srv409094444=20, srv218440738=16, srv480185491=22, srv2024915786=12, srv415866164=21, srv38281468=19, srv197745920=11, srv489756226=23, srv877766955=28, srv2071963710=13, srv348300747=17, srv1969155229=9, srv605596685=24, srv1299310774=1, srv2146319371=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-06 15:57:39,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-06 15:57:39,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-06 15:57:39,992 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-06 15:57:39,992 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1194705503:90 , srv1299310774:90 , srv1363486183:90 , srv1468222236:90 , srv1621590261:90 , srv1661627138:90 , srv1743782673:90 , srv1761126025:90 , srv1839257084:90 , srv1969155229:90 , srv1973712768:90 , srv197745920:90 , srv2024915786:90 , srv2071963710:90 , srv2127818193:90 , srv2146319371:90 , srv218440738:90 , srv348300747:90 , srv374311070:90 , srv38281468:90 , srv409094444:90 , srv415866164:90 , srv480185491:90 , srv489756226:90 , srv605596685:90 , srv708514584:90 , srv762447764:90 , srv797661327:90 , srv877766955:90 , srv990184919:90 } 2023-11-06 15:57:40,003 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=164 (was 200), ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=14786 (was 15003)