2023-09-21 06:53:14,576 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-21 06:53:14,901 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-21 06:53:15,013 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-21 06:53:15,081 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-21 06:53:15,091 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-21 06:53:15,105 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=170, AvailableMemoryMB=4228 2023-09-21 06:53:15,107 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-21 06:53:15,108 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-21 06:53:15,259 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010654778:84 , srv1035335695:84 , srv1039797589:84 , srv1044122671:84 , srv1083497788:84 , srv1091252818:84 , srv1116733201:84 , srv1144613024:84 , srv1149628796:84 , srv1150020155:84 , srv1178154007:84 , srv1194686666:84 , srv1195056175:84 , srv1223864295:84 , srv123176798:84 , srv1254860010:84 , srv1256752752:84 , srv126239944:84 , srv1282503634:84 , srv1285803840:84 , srv1288743620:84 , srv1289235476:84 , srv1289262139:84 , srv1309064392:84 , srv1319121291:84 , srv134157632:84 , srv1434813262:84 , srv1437945210:84 , srv1453764378:84 , srv1536331101:84 , srv1547798632:84 , srv1552032708:84 , srv1561001800:84 , srv1568726696:84 , srv1573417332:84 , srv1574205958:84 , srv1579586506:84 , srv1640980443:84 , srv1661493346:84 , srv1675099986:84 , srv1733729265:84 , srv1738009514:84 , srv1744389809:84 , srv1828008921:84 , srv1865824063:84 , srv1877224178:84 , srv1878054323:84 , srv1888021366:84 , srv1889953572:84 , srv1890806468:84 , srv1892763965:84 , srv1908283044:84 , srv1933031338:84 , srv196768833:84 , srv1974331742:84 , srv1983411520:84 , srv2004550450:84 , srv2032602866:84 , srv205657033:84 , srv207114849:84 , srv2096613709:84 , srv2100087189:84 , srv2112882277:84 , srv2121927435:84 , srv238970979:84 , srv254659478:84 , srv255643411:84 , srv262796029:84 , srv284230713:84 , srv289258682:84 , srv348534414:84 , srv359197656:84 , srv436475913:84 , srv447125345:84 , srv453113719:84 , srv505551796:84 , srv58147395:84 , srv585454669:84 , srv609068768:84 , srv610225085:84 , srv627761045:84 , srv685852635:84 , srv697103097:84 , srv701330940:84 , srv703674098:84 , srv707891643:84 , srv735503006:84 , srv736605850:84 , srv756282653:84 , srv808659585:84 , srv827565055:84 , srv829037320:84 , srv846827558:84 , srv847971008:84 , srv849276382:84 , srv861199331:84 , srv874196344:84 , srv878316709:84 , srv923519213:84 , srv1897819992:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-21 06:53:15,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:53:15,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1039797589=2, srv1888021366=47, srv1150020155=9, srv1309064392=23, srv609068768=79, srv1536331101=29, srv1573417332=34, srv196768833=54, srv685852635=82, srv874196344=97, srv1878054323=46, srv1254860010=15, srv1733729265=40, srv1285803840=19, srv1890806468=49, srv827565055=91, srv1983411520=56, srv878316709=98, srv289258682=70, srv846827558=93, srv703674098=85, srv1574205958=35, srv1437945210=27, srv505551796=76, srv1579586506=36, srv1828008921=43, srv829037320=92, srv1738009514=41, srv2004550450=57, srv2096613709=61, srv610225085=80, srv1889953572=48, srv123176798=14, srv262796029=68, srv2112882277=63, srv284230713=69, srv861199331=96, srv1256752752=16, srv707891643=86, srv701330940=84, srv1675099986=39, srv205657033=59, srv348534414=71, srv1289262139=22, srv585454669=78, srv359197656=72, srv697103097=83, srv1933031338=53, srv58147395=77, srv255643411=67, srv134157632=25, srv1908283044=52, srv1144613024=7, srv2032602866=58, srv1195056175=12, srv1289235476=21, srv2100087189=62, srv1434813262=26, srv254659478=66, srv735503006=87, srv436475913=73, srv1083497788=4, srv1044122671=3, srv627761045=81, srv1288743620=20, srv1010654778=0, srv453113719=75, srv1035335695=1, srv447125345=74, srv1194686666=11, srv1223864295=13, srv808659585=90, srv1547798632=30, srv1149628796=8, srv1974331742=55, srv1640980443=37, srv1282503634=18, srv1453764378=28, srv1552032708=31, srv849276382=95, srv756282653=89, srv1116733201=6, srv1091252818=5, srv1892763965=50, srv1661493346=38, srv126239944=17, srv847971008=94, srv923519213=99, srv1865824063=44, srv1561001800=32, srv1897819992=51, srv2121927435=64, srv1744389809=42, srv1178154007=10, srv1877224178=45, srv1568726696=33, srv736605850=88, srv238970979=65, srv1319121291=24, srv207114849=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:53:15,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:53:15,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-21 06:53:15,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-21 06:53:15,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-21 06:53:15,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-21 06:53:15,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-21 06:53:15,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-21 06:53:15,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-21 06:53:15,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-21 06:53:15,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-21 06:53:15,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-21 06:53:15,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-21 06:53:15,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-21 06:53:15,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-21 06:53:15,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-21 06:53:15,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-21 06:53:15,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-21 06:53:15,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-21 06:53:15,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-21 06:53:15,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-21 06:53:15,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-21 06:53:15,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-21 06:53:15,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-21 06:53:15,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-21 06:53:15,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-21 06:53:15,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-21 06:53:15,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:15,669 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-21 06:53:15,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-21 06:53:16,549 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1122 ms to try 5000 different iterations. Found a solution that moves 4759 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17174503939323219. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03967035888087444, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05233064180601461, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5287777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03967035888087444, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1665, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23758333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:16,589 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1889953572:81 , srv1890806468:83 , srv262796029:83 , srv735503006:83 , srv1675099986:84 , srv289258682:84 , srv697103097:84 , srv707891643:84 , srv808659585:84 , srv827565055:84 , srv874196344:84 , srv1039797589:85 , srv1091252818:85 , srv1144613024:85 , srv1254860010:85 , srv1285803840:85 , srv1288743620:85 , srv1319121291:85 , srv1574205958:85 , srv1738009514:85 , srv1744389809:85 , srv1878054323:85 , srv1908283044:85 , srv1983411520:85 , srv254659478:85 , srv703674098:85 , srv849276382:85 , srv878316709:85 , srv923519213:85 , srv1035335695:86 , srv1116733201:86 , srv1178154007:86 , srv1195056175:86 , srv1223864295:86 , srv1282503634:86 , srv1289262139:86 , srv1309064392:86 , srv134157632:86 , srv1568726696:86 , srv1865824063:86 , srv1888021366:86 , srv196768833:86 , srv2032602866:86 , srv207114849:86 , srv2096613709:86 , srv2112882277:86 , srv238970979:86 , srv447125345:86 , srv505551796:86 , srv585454669:86 , srv627761045:86 , srv701330940:86 , srv829037320:86 , srv846827558:86 , srv1044122671:87 , srv1083497788:87 , srv1194686666:87 , srv123176798:87 , srv1289235476:87 , srv1434813262:87 , srv1547798632:87 , srv1561001800:87 , srv1579586506:87 , srv1640980443:87 , srv1661493346:87 , srv1733729265:87 , srv1828008921:87 , srv1877224178:87 , srv1933031338:87 , srv1974331742:87 , srv205657033:87 , srv2121927435:87 , srv284230713:87 , srv348534414:87 , srv685852635:87 , srv736605850:87 , srv861199331:87 , srv1010654778:88 , srv1150020155:88 , srv126239944:88 , srv1536331101:88 , srv1552032708:88 , srv1573417332:88 , srv1892763965:88 , srv255643411:88 , srv359197656:88 , srv756282653:88 , srv1149628796:89 , srv1256752752:89 , srv1437945210:89 , srv2100087189:89 , srv436475913:89 , srv58147395:89 , srv2004550450:90 , srv609068768:90 , srv847971008:90 , srv453113719:91 , srv610225085:91 , srv1453764378:92 , srv1897819992:443 } 2023-09-21 06:53:16,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:53:16,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1039797589=2, srv1888021366=47, srv1150020155=9, srv1309064392=23, srv609068768=79, srv1536331101=29, srv1573417332=34, srv196768833=54, srv685852635=82, srv874196344=97, srv1878054323=46, srv1254860010=15, srv1733729265=40, srv1285803840=19, srv1890806468=49, srv827565055=91, srv1983411520=56, srv878316709=98, srv289258682=70, srv846827558=93, srv703674098=85, srv1574205958=35, srv1437945210=27, srv505551796=76, srv1579586506=36, srv1828008921=43, srv829037320=92, srv1738009514=41, srv2004550450=57, srv2096613709=61, srv610225085=80, srv1889953572=48, srv123176798=14, srv262796029=68, srv2112882277=63, srv284230713=69, srv861199331=96, srv1256752752=16, srv707891643=86, srv701330940=84, srv1675099986=39, srv205657033=59, srv348534414=71, srv1289262139=22, srv585454669=78, srv359197656=72, srv697103097=83, srv1933031338=53, srv58147395=77, srv255643411=67, srv134157632=25, srv1908283044=52, srv1144613024=7, srv2032602866=58, srv1195056175=12, srv1289235476=21, srv2100087189=62, srv1434813262=26, srv254659478=66, srv735503006=87, srv436475913=73, srv1083497788=4, srv1044122671=3, srv627761045=81, srv1288743620=20, srv1010654778=0, srv453113719=75, srv1035335695=1, srv447125345=74, srv1194686666=11, srv1223864295=13, srv808659585=90, srv1547798632=30, srv1149628796=8, srv1974331742=55, srv1640980443=37, srv1282503634=18, srv1453764378=28, srv1552032708=31, srv849276382=95, srv756282653=89, srv1116733201=6, srv1091252818=5, srv1892763965=50, srv1661493346=38, srv126239944=17, srv847971008=94, srv923519213=99, srv1865824063=44, srv1561001800=32, srv1897819992=51, srv2121927435=64, srv1744389809=42, srv1178154007=10, srv1877224178=45, srv1568726696=33, srv736605850=88, srv238970979=65, srv1319121291=24, srv207114849=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-21 06:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-21 06:53:16,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-21 06:53:16,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-21 06:53:16,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-21 06:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-21 06:53:16,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-21 06:53:16,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-21 06:53:16,893 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03967035888087444, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05233064180601461, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03967035888087444, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1665, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23758333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:16,893 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-21 06:53:16,894 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17171171616300052, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03967035888087444, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05233064180601461, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03967035888087444, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1665, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23758333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-21 06:53:17,269 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 609 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.17171171616300052 to a new imbalance of 0.01458228488948942. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021717615426260956); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03221351327797758, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021717615426260956); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:17,284 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1453764378:79 , srv453113719:79 , srv2100087189:81 , srv1552032708:82 , srv1573417332:82 , srv436475913:82 , srv610225085:82 , srv1010654778:83 , srv1044122671:83 , srv1150020155:83 , srv126239944:83 , srv1437945210:83 , srv2004550450:83 , srv505551796:83 , srv609068768:83 , srv1083497788:84 , srv1149628796:84 , srv123176798:84 , srv1536331101:84 , srv1547798632:84 , srv1561001800:84 , srv1640980443:84 , srv1738009514:84 , srv1828008921:84 , srv196768833:84 , srv1974331742:84 , srv2096613709:84 , srv284230713:84 , srv359197656:84 , srv701330940:84 , srv1116733201:85 , srv1178154007:85 , srv1223864295:85 , srv1256752752:85 , srv1288743620:85 , srv1661493346:85 , srv1865824063:85 , srv1877224178:85 , srv1878054323:85 , srv1888021366:85 , srv1892763965:85 , srv2032602866:85 , srv58147395:85 , srv685852635:85 , srv756282653:85 , srv846827558:85 , srv847971008:85 , srv861199331:85 , srv1035335695:86 , srv1195056175:86 , srv1289235476:86 , srv1289262139:86 , srv1309064392:86 , srv1434813262:86 , srv1568726696:86 , srv1579586506:86 , srv205657033:86 , srv207114849:86 , srv2121927435:86 , srv255643411:86 , srv348534414:86 , srv447125345:86 , srv736605850:86 , srv829037320:86 , srv1254860010:87 , srv1319121291:87 , srv134157632:87 , srv1574205958:87 , srv1890806468:87 , srv1908283044:87 , srv1933031338:87 , srv2112882277:87 , srv289258682:87 , srv585454669:87 , srv627761045:87 , srv703674098:87 , srv827565055:87 , srv1144613024:88 , srv1194686666:88 , srv1285803840:88 , srv1733729265:88 , srv1744389809:88 , srv1889953572:88 , srv1983411520:88 , srv238970979:88 , srv697103097:88 , srv707891643:88 , srv808659585:88 , srv923519213:88 , srv1039797589:89 , srv1091252818:89 , srv1282503634:89 , srv1675099986:89 , srv254659478:89 , srv735503006:89 , srv849276382:89 , srv878316709:89 , srv262796029:90 , srv874196344:90 , srv1897819992:524 } 2023-09-21 06:53:17,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:53:17,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1039797589=2, srv1888021366=47, srv1150020155=9, srv1309064392=23, srv609068768=79, srv1536331101=29, srv1573417332=34, srv196768833=54, srv685852635=82, srv874196344=97, srv1878054323=46, srv1254860010=15, srv1733729265=40, srv1285803840=19, srv1890806468=49, srv827565055=91, srv1983411520=56, srv878316709=98, srv289258682=70, srv846827558=93, srv703674098=85, srv1574205958=35, srv1437945210=27, srv505551796=76, srv1579586506=36, srv1828008921=43, srv829037320=92, srv1738009514=41, srv2004550450=57, srv2096613709=61, srv610225085=80, srv1889953572=48, srv123176798=14, srv262796029=68, srv2112882277=63, srv284230713=69, srv861199331=96, srv1256752752=16, srv707891643=86, srv701330940=84, srv1675099986=39, srv205657033=59, srv348534414=71, srv1289262139=22, srv585454669=78, srv359197656=72, srv697103097=83, srv1933031338=53, srv58147395=77, srv255643411=67, srv134157632=25, srv1908283044=52, srv1144613024=7, srv2032602866=58, srv1195056175=12, srv1289235476=21, srv2100087189=62, srv1434813262=26, srv254659478=66, srv735503006=87, srv436475913=73, srv1083497788=4, srv1044122671=3, srv627761045=81, srv1288743620=20, srv1010654778=0, srv453113719=75, srv1035335695=1, srv447125345=74, srv1194686666=11, srv1223864295=13, srv808659585=90, srv1547798632=30, srv1149628796=8, srv1974331742=55, srv1640980443=37, srv1282503634=18, srv1453764378=28, srv1552032708=31, srv849276382=95, srv756282653=89, srv1116733201=6, srv1091252818=5, srv1892763965=50, srv1661493346=38, srv126239944=17, srv847971008=94, srv923519213=99, srv1865824063=44, srv1561001800=32, srv1897819992=51, srv2121927435=64, srv1744389809=42, srv1178154007=10, srv1877224178=45, srv1568726696=33, srv736605850=88, srv238970979=65, srv1319121291=24, srv207114849=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:53:17,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:53:17,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:53:17,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:53:17,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:53:17,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:53:17,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:53:17,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:53:17,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-21 06:53:17,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-21 06:53:17,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-21 06:53:17,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-21 06:53:17,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-21 06:53:17,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-21 06:53:17,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-21 06:53:17,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-21 06:53:17,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-21 06:53:17,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-21 06:53:17,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-21 06:53:17,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-21 06:53:17,515 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021717615426260956); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03221351327797758, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021717615426260956); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:17,515 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-21 06:53:17,515 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0145640022767273, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021717615426260956); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03221351327797758, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021717615426260956); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-21 06:53:17,730 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 338 ms to try 5000 different iterations. Found a solution that moves 955 regions; Going from a computed imbalance of 0.0145640022767273 to a new imbalance of 6.058649919046065E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00588156798823659); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006816498108607268); MoveCostFunction : (multiplier=7.0, imbalance=0.10611111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00588156798823659); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:17,731 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1194686666:82 , srv1733729265:82 , srv847971008:82 , srv134157632:83 , srv1933031338:83 , srv254659478:83 , srv255643411:83 , srv849276382:83 , srv874196344:83 , srv1091252818:84 , srv1144613024:84 , srv1282503634:84 , srv1285803840:84 , srv1675099986:84 , srv1744389809:84 , srv1892763965:84 , srv2004550450:84 , srv238970979:84 , srv348534414:84 , srv58147395:84 , srv609068768:84 , srv878316709:84 , srv923519213:84 , srv1035335695:85 , srv1039797589:85 , srv1149628796:85 , srv1195056175:85 , srv1256752752:85 , srv1289235476:85 , srv1309064392:85 , srv1319121291:85 , srv1434813262:85 , srv1437945210:85 , srv1536331101:85 , srv1574205958:85 , srv1579586506:85 , srv1661493346:85 , srv1908283044:85 , srv205657033:85 , srv207114849:85 , srv2112882277:85 , srv2121927435:85 , srv262796029:85 , srv447125345:85 , srv585454669:85 , srv627761045:85 , srv685852635:85 , srv697103097:85 , srv703674098:85 , srv735503006:85 , srv1150020155:86 , srv1178154007:86 , srv1223864295:86 , srv1254860010:86 , srv126239944:86 , srv1289262139:86 , srv1547798632:86 , srv1561001800:86 , srv1568726696:86 , srv1828008921:86 , srv1865824063:86 , srv1877224178:86 , srv1974331742:86 , srv1983411520:86 , srv2032602866:86 , srv284230713:86 , srv359197656:86 , srv436475913:86 , srv610225085:86 , srv707891643:86 , srv736605850:86 , srv756282653:86 , srv808659585:86 , srv829037320:86 , srv846827558:86 , srv861199331:86 , srv1010654778:87 , srv1044122671:87 , srv1083497788:87 , srv1116733201:87 , srv123176798:87 , srv1288743620:87 , srv1453764378:87 , srv1573417332:87 , srv1640980443:87 , srv1878054323:87 , srv1888021366:87 , srv196768833:87 , srv2096613709:87 , srv2100087189:87 , srv289258682:87 , srv453113719:87 , srv505551796:87 , srv701330940:87 , srv827565055:87 , srv1552032708:88 , srv1889953572:88 , srv1890806468:88 , srv1738009514:89 , srv1897819992:543 } 2023-09-21 06:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:53:17,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1039797589=2, srv1888021366=47, srv1150020155=9, srv1309064392=23, srv609068768=79, srv1536331101=29, srv1573417332=34, srv196768833=54, srv685852635=82, srv874196344=97, srv1878054323=46, srv1254860010=15, srv1733729265=40, srv1285803840=19, srv1890806468=49, srv827565055=91, srv1983411520=56, srv878316709=98, srv289258682=70, srv846827558=93, srv703674098=85, srv1574205958=35, srv1437945210=27, srv505551796=76, srv1579586506=36, srv1828008921=43, srv829037320=92, srv1738009514=41, srv2004550450=57, srv2096613709=61, srv610225085=80, srv1889953572=48, srv123176798=14, srv262796029=68, srv2112882277=63, srv284230713=69, srv861199331=96, srv1256752752=16, srv707891643=86, srv701330940=84, srv1675099986=39, srv205657033=59, srv348534414=71, srv1289262139=22, srv585454669=78, srv359197656=72, srv697103097=83, srv1933031338=53, srv58147395=77, srv255643411=67, srv134157632=25, srv1908283044=52, srv1144613024=7, srv2032602866=58, srv1195056175=12, srv1289235476=21, srv2100087189=62, srv1434813262=26, srv254659478=66, srv735503006=87, srv436475913=73, srv1083497788=4, srv1044122671=3, srv627761045=81, srv1288743620=20, srv1010654778=0, srv453113719=75, srv1035335695=1, srv447125345=74, srv1194686666=11, srv1223864295=13, srv808659585=90, srv1547798632=30, srv1149628796=8, srv1974331742=55, srv1640980443=37, srv1282503634=18, srv1453764378=28, srv1552032708=31, srv849276382=95, srv756282653=89, srv1116733201=6, srv1091252818=5, srv1892763965=50, srv1661493346=38, srv126239944=17, srv847971008=94, srv923519213=99, srv1865824063=44, srv1561001800=32, srv1897819992=51, srv2121927435=64, srv1744389809=42, srv1178154007=10, srv1877224178=45, srv1568726696=33, srv736605850=88, srv238970979=65, srv1319121291=24, srv207114849=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-21 06:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-21 06:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-21 06:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-21 06:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-21 06:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-21 06:53:17,866 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00588156798823659); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006816498108607268); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00588156798823659); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:17,866 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-21 06:53:17,867 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.991779389793585E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00588156798823659); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006816498108607268); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00588156798823659); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-21 06:53:18,184 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 422 ms to try 5000 different iterations. Found a solution that moves 234 regions; Going from a computed imbalance of 5.991779389793585E-4 to a new imbalance of 1.2837077888946126E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.026, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:18,185 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv756282653:82 , srv1039797589:83 , srv205657033:83 , srv262796029:83 , srv610225085:83 , srv627761045:83 , srv736605850:83 , srv829037320:83 , srv1010654778:84 , srv1083497788:84 , srv1091252818:84 , srv1149628796:84 , srv1150020155:84 , srv1194686666:84 , srv123176798:84 , srv1254860010:84 , srv1256752752:84 , srv1282503634:84 , srv1289235476:84 , srv1289262139:84 , srv1453764378:84 , srv1552032708:84 , srv1568726696:84 , srv1579586506:84 , srv1738009514:84 , srv1877224178:84 , srv1888021366:84 , srv1974331742:84 , srv1983411520:84 , srv2112882277:84 , srv2121927435:84 , srv238970979:84 , srv255643411:84 , srv289258682:84 , srv359197656:84 , srv585454669:84 , srv707891643:84 , srv827565055:84 , srv849276382:84 , srv861199331:84 , srv878316709:84 , srv1035335695:85 , srv1044122671:85 , srv1116733201:85 , srv1144613024:85 , srv1178154007:85 , srv1195056175:85 , srv1223864295:85 , srv126239944:85 , srv1285803840:85 , srv1288743620:85 , srv1309064392:85 , srv1319121291:85 , srv1434813262:85 , srv1437945210:85 , srv1536331101:85 , srv1561001800:85 , srv1573417332:85 , srv1574205958:85 , srv1640980443:85 , srv1661493346:85 , srv1675099986:85 , srv1733729265:85 , srv1744389809:85 , srv1828008921:85 , srv1865824063:85 , srv1878054323:85 , srv1889953572:85 , srv1890806468:85 , srv1892763965:85 , srv1908283044:85 , srv1933031338:85 , srv196768833:85 , srv2004550450:85 , srv2032602866:85 , srv207114849:85 , srv2096613709:85 , srv2100087189:85 , srv254659478:85 , srv284230713:85 , srv348534414:85 , srv436475913:85 , srv447125345:85 , srv453113719:85 , srv58147395:85 , srv609068768:85 , srv685852635:85 , srv697103097:85 , srv701330940:85 , srv703674098:85 , srv735503006:85 , srv808659585:85 , srv846827558:85 , srv847971008:85 , srv874196344:85 , srv923519213:85 , srv134157632:86 , srv1547798632:86 , srv505551796:86 , srv1897819992:632 } 2023-09-21 06:53:18,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:53:18,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1039797589=2, srv1888021366=47, srv1150020155=9, srv1309064392=23, srv609068768=79, srv1536331101=29, srv1573417332=34, srv196768833=54, srv685852635=82, srv874196344=97, srv1878054323=46, srv1254860010=15, srv1733729265=40, srv1285803840=19, srv1890806468=49, srv827565055=91, srv1983411520=56, srv878316709=98, srv289258682=70, srv846827558=93, srv703674098=85, srv1574205958=35, srv1437945210=27, srv505551796=76, srv1579586506=36, srv1828008921=43, srv829037320=92, srv1738009514=41, srv2004550450=57, srv2096613709=61, srv610225085=80, srv1889953572=48, srv123176798=14, srv262796029=68, srv2112882277=63, srv284230713=69, srv861199331=96, srv1256752752=16, srv707891643=86, srv701330940=84, srv1675099986=39, srv205657033=59, srv348534414=71, srv1289262139=22, srv585454669=78, srv359197656=72, srv697103097=83, srv1933031338=53, srv58147395=77, srv255643411=67, srv134157632=25, srv1908283044=52, srv1144613024=7, srv2032602866=58, srv1195056175=12, srv1289235476=21, srv2100087189=62, srv1434813262=26, srv254659478=66, srv735503006=87, srv436475913=73, srv1083497788=4, srv1044122671=3, srv627761045=81, srv1288743620=20, srv1010654778=0, srv453113719=75, srv1035335695=1, srv447125345=74, srv1194686666=11, srv1223864295=13, srv808659585=90, srv1547798632=30, srv1149628796=8, srv1974331742=55, srv1640980443=37, srv1282503634=18, srv1453764378=28, srv1552032708=31, srv849276382=95, srv756282653=89, srv1116733201=6, srv1091252818=5, srv1892763965=50, srv1661493346=38, srv126239944=17, srv847971008=94, srv923519213=99, srv1865824063=44, srv1561001800=32, srv1897819992=51, srv2121927435=64, srv1744389809=42, srv1178154007=10, srv1877224178=45, srv1568726696=33, srv736605850=88, srv238970979=65, srv1319121291=24, srv207114849=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:53:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:53:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-21 06:53:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-21 06:53:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-21 06:53:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-21 06:53:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-21 06:53:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-21 06:53:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-21 06:53:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-21 06:53:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-21 06:53:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-21 06:53:18,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-21 06:53:18,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-21 06:53:18,325 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1198574868518857E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:18,326 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv756282653:82 , srv1039797589:83 , srv205657033:83 , srv262796029:83 , srv610225085:83 , srv627761045:83 , srv736605850:83 , srv829037320:83 , srv1010654778:84 , srv1083497788:84 , srv1091252818:84 , srv1149628796:84 , srv1150020155:84 , srv1194686666:84 , srv123176798:84 , srv1254860010:84 , srv1256752752:84 , srv1282503634:84 , srv1289235476:84 , srv1289262139:84 , srv1453764378:84 , srv1552032708:84 , srv1568726696:84 , srv1579586506:84 , srv1738009514:84 , srv1877224178:84 , srv1888021366:84 , srv1974331742:84 , srv1983411520:84 , srv2112882277:84 , srv2121927435:84 , srv238970979:84 , srv255643411:84 , srv289258682:84 , srv359197656:84 , srv585454669:84 , srv707891643:84 , srv827565055:84 , srv849276382:84 , srv861199331:84 , srv878316709:84 , srv1035335695:85 , srv1044122671:85 , srv1116733201:85 , srv1144613024:85 , srv1178154007:85 , srv1195056175:85 , srv1223864295:85 , srv126239944:85 , srv1285803840:85 , srv1288743620:85 , srv1309064392:85 , srv1319121291:85 , srv1434813262:85 , srv1437945210:85 , srv1536331101:85 , srv1561001800:85 , srv1573417332:85 , srv1574205958:85 , srv1640980443:85 , srv1661493346:85 , srv1675099986:85 , srv1733729265:85 , srv1744389809:85 , srv1828008921:85 , srv1865824063:85 , srv1878054323:85 , srv1889953572:85 , srv1890806468:85 , srv1892763965:85 , srv1908283044:85 , srv1933031338:85 , srv196768833:85 , srv2004550450:85 , srv2032602866:85 , srv207114849:85 , srv2096613709:85 , srv2100087189:85 , srv254659478:85 , srv284230713:85 , srv348534414:85 , srv436475913:85 , srv447125345:85 , srv453113719:85 , srv58147395:85 , srv609068768:85 , srv685852635:85 , srv697103097:85 , srv701330940:85 , srv703674098:85 , srv735503006:85 , srv808659585:85 , srv846827558:85 , srv847971008:85 , srv874196344:85 , srv923519213:85 , srv134157632:86 , srv1547798632:86 , srv505551796:86 , srv1897819992:632 } 2023-09-21 06:53:18,354 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=333 (was 301) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=3620 (was 4228) 2023-09-21 06:53:18,362 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=170, AvailableMemoryMB=3620 2023-09-21 06:53:18,363 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-21 06:53:18,363 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-21 06:53:18,370 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1057289205:84 , srv1147323367:84 , srv148059446:84 , srv1604256493:84 , srv1617397543:84 , srv1653548606:84 , srv1692950682:84 , srv193870871:84 , srv199549583:84 , srv2092689368:84 , srv215066375:84 , srv231371246:84 , srv285690305:84 , srv317047257:84 , srv32284355:84 , srv328738839:84 , srv345665571:84 , srv385038794:84 , srv427442447:84 , srv427482392:84 , srv450896479:84 , srv534331334:84 , srv55781267:84 , srv705118438:84 , srv792115678:84 , srv814429200:84 , srv833969196:84 , srv909276566:84 , srv962631640:84 , srv808766700:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-21 06:53:18,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:53:18,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1057289205=0, srv962631640=29, srv909276566=28, srv1147323367=1, srv1604256493=3, srv285690305=12, srv1692950682=6, srv55781267=22, srv2092689368=9, srv193870871=7, srv317047257=13, srv534331334=21, srv199549583=8, srv345665571=16, srv328738839=15, srv1653548606=5, srv705118438=23, srv833969196=27, srv148059446=2, srv427442447=18, srv450896479=20, srv385038794=17, srv792115678=24, srv231371246=11, srv808766700=25, srv1617397543=4, srv32284355=14, srv814429200=26, srv215066375=10, srv427482392=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-21 06:53:18,396 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1156907424461344, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:53:18,396 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912557581016077, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1156907424461344, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-21 06:55:18,384 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6383519 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912557581016077 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:55:18,391 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1057289205:90 , srv1147323367:90 , srv148059446:90 , srv1604256493:90 , srv1617397543:90 , srv1653548606:90 , srv1692950682:90 , srv193870871:90 , srv199549583:90 , srv2092689368:90 , srv215066375:90 , srv231371246:90 , srv285690305:90 , srv317047257:90 , srv32284355:90 , srv328738839:90 , srv345665571:90 , srv385038794:90 , srv427442447:90 , srv427482392:90 , srv450896479:90 , srv534331334:90 , srv55781267:90 , srv705118438:90 , srv792115678:90 , srv808766700:90 , srv814429200:90 , srv833969196:90 , srv909276566:90 , srv962631640:90 } 2023-09-21 06:55:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-21 06:55:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1057289205=0, srv962631640=29, srv909276566=28, srv1147323367=1, srv1604256493=3, srv285690305=12, srv1692950682=6, srv55781267=22, srv2092689368=9, srv193870871=7, srv317047257=13, srv534331334=21, srv199549583=8, srv345665571=16, srv328738839=15, srv1653548606=5, srv705118438=23, srv833969196=27, srv148059446=2, srv427442447=18, srv450896479=20, srv385038794=17, srv792115678=24, srv231371246=11, srv808766700=25, srv1617397543=4, srv32284355=14, srv814429200=26, srv215066375=10, srv427482392=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-21 06:55:18,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-21 06:55:18,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-21 06:55:18,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-21 06:55:18,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-21 06:55:18,414 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1057289205:90 , srv1147323367:90 , srv148059446:90 , srv1604256493:90 , srv1617397543:90 , srv1653548606:90 , srv1692950682:90 , srv193870871:90 , srv199549583:90 , srv2092689368:90 , srv215066375:90 , srv231371246:90 , srv285690305:90 , srv317047257:90 , srv32284355:90 , srv328738839:90 , srv345665571:90 , srv385038794:90 , srv427442447:90 , srv427482392:90 , srv450896479:90 , srv534331334:90 , srv55781267:90 , srv705118438:90 , srv792115678:90 , srv808766700:90 , srv814429200:90 , srv833969196:90 , srv909276566:90 , srv962631640:90 } 2023-09-21 06:55:18,427 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 333), ProcessCount=170 (was 170), AvailableMemoryMB=2652 (was 3620)