2023-03-14 20:54:17,900 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-14 20:54:18,211 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-14 20:54:18,322 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-14 20:54:18,382 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-14 20:54:18,389 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-14 20:54:18,401 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=169, AvailableMemoryMB=3742 2023-03-14 20:54:18,403 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-14 20:54:18,404 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-14 20:54:18,522 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009740286:84 , srv1083060887:84 , srv1165792890:84 , srv1169252771:84 , srv1252518198:84 , srv1275856101:84 , srv1279630480:84 , srv1280673857:84 , srv1290738472:84 , srv1299990919:84 , srv1306781648:84 , srv1380213936:84 , srv1384888874:84 , srv1397496143:84 , srv1414665857:84 , srv1432411076:84 , srv1471229510:84 , srv1489501786:84 , srv1494696149:84 , srv1501977562:84 , srv1519784983:84 , srv1528466002:84 , srv1530046590:84 , srv1533162100:84 , srv1538198283:84 , srv1554849321:84 , srv1566582908:84 , srv1618989914:84 , srv1641596955:84 , srv1647632491:84 , srv1650431426:84 , srv1656369887:84 , srv1720222672:84 , srv1722042277:84 , srv1747927134:84 , srv1763241379:84 , srv1795980542:84 , srv1796912734:84 , srv1803007052:84 , srv1832965156:84 , srv1839252278:84 , srv1842800908:84 , srv1868473963:84 , srv1875179654:84 , srv187832191:84 , srv1894726413:84 , srv1952371182:84 , srv1988510570:84 , srv1991450488:84 , srv2034108042:84 , srv2068058947:84 , srv2078107980:84 , srv2114091763:84 , srv2119831386:84 , srv2128080621:84 , srv217759238:84 , srv240242219:84 , srv255378570:84 , srv330215771:84 , srv33953718:84 , srv346142494:84 , srv366595783:84 , srv414508150:84 , srv42081658:84 , srv42680951:84 , srv440330932:84 , srv443665382:84 , srv444913462:84 , srv446753645:84 , srv4597393:84 , srv499240490:84 , srv502433346:84 , srv511570336:84 , srv515292452:84 , srv523207130:84 , srv556154365:84 , srv619910803:84 , srv636703763:84 , srv655949381:84 , srv672257538:84 , srv680007282:84 , srv696437562:84 , srv705568401:84 , srv721835014:84 , srv756555871:84 , srv788335463:84 , srv79864763:84 , srv80210886:84 , srv803495285:84 , srv82642247:84 , srv8403639:84 , srv844945477:84 , srv846765651:84 , srv855938919:84 , srv915688649:84 , srv927190707:84 , srv929795704:84 , srv953291038:84 , srv996052487:84 , srv1909685667:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-14 20:54:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33953718=60, srv1432411076=15, srv523207130=75, srv1252518198=4, srv1554849321=25, srv443665382=67, srv1720222672=32, srv953291038=98, srv1650431426=30, srv927190707=96, srv705568401=83, srv440330932=66, srv2119831386=54, srv414508150=63, srv1528466002=21, srv42680951=65, srv1988510570=48, srv929795704=97, srv1519784983=20, srv1280673857=7, srv2078107980=52, srv855938919=94, srv187832191=44, srv721835014=84, srv680007282=81, srv80210886=88, srv330215771=59, srv1538198283=24, srv1722042277=33, srv619910803=77, srv2128080621=55, srv1795980542=36, srv1299990919=9, srv1641596955=28, srv1290738472=8, srv42081658=64, srv1083060887=1, srv1275856101=5, srv366595783=62, srv4597393=70, srv240242219=57, srv1306781648=10, srv1842800908=41, srv636703763=78, srv1489501786=17, srv1656369887=31, srv1494696149=18, srv1165792890=2, srv1868473963=42, srv444913462=68, srv1875179654=43, srv1803007052=38, srv1747927134=34, srv1566582908=26, srv756555871=85, srv499240490=71, srv1279630480=6, srv217759238=56, srv515292452=74, srv1530046590=22, srv2114091763=53, srv79864763=87, srv1501977562=19, srv346142494=61, srv655949381=79, srv996052487=99, srv2034108042=50, srv1647632491=29, srv446753645=69, srv82642247=90, srv1380213936=11, srv1832965156=39, srv803495285=89, srv844945477=92, srv846765651=93, srv1397496143=13, srv1952371182=47, srv696437562=82, srv556154365=76, srv1471229510=16, srv2068058947=51, srv1533162100=23, srv788335463=86, srv1169252771=3, srv672257538=80, srv1796912734=37, srv915688649=95, srv1991450488=49, srv1384888874=12, srv511570336=73, srv1763241379=35, srv1414665857=14, srv1618989914=27, srv1894726413=45, srv1909685667=46, srv8403639=91, srv1009740286=0, srv1839252278=40, srv255378570=58, srv502433346=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:18,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:18,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:18,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:18,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:18,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:18,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:18,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:18,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:18,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-14 20:54:18,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-14 20:54:18,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-14 20:54:18,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-14 20:54:18,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-14 20:54:18,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-14 20:54:18,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-14 20:54:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-14 20:54:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-14 20:54:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-14 20:54:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-14 20:54:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-14 20:54:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-14 20:54:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-14 20:54:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-14 20:54:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-14 20:54:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-14 20:54:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-14 20:54:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-14 20:54:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-14 20:54:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-14 20:54:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-14 20:54:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-14 20:54:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-14 20:54:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-14 20:54:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-14 20:54:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-14 20:54:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-14 20:54:18,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:18,826 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-14 20:54:18,826 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-14 20:54:19,193 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 518 ms to try 5000 different iterations. Found a solution that moves 4712 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17870981572692363. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04396997013984675, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06005609050491054, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5235555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04396997013984675, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:19,204 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv929795704:81 , srv1533162100:82 , srv1763241379:82 , srv1839252278:82 , srv499240490:82 , srv619910803:82 , srv1083060887:83 , srv1299990919:83 , srv1384888874:83 , srv1554849321:83 , srv1803007052:83 , srv1988510570:83 , srv2068058947:83 , srv33953718:83 , srv502433346:83 , srv523207130:83 , srv79864763:83 , srv1169252771:84 , srv1252518198:84 , srv1306781648:84 , srv1397496143:84 , srv1414665857:84 , srv1747927134:84 , srv1875179654:84 , srv217759238:84 , srv346142494:84 , srv443665382:84 , srv672257538:84 , srv927190707:84 , srv1275856101:85 , srv1280673857:85 , srv1432411076:85 , srv1720222672:85 , srv1722042277:85 , srv1842800908:85 , srv2078107980:85 , srv42081658:85 , srv42680951:85 , srv444913462:85 , srv4597393:85 , srv515292452:85 , srv721835014:85 , srv996052487:85 , srv1009740286:86 , srv1279630480:86 , srv1380213936:86 , srv1494696149:86 , srv1501977562:86 , srv1528466002:86 , srv1641596955:86 , srv1656369887:86 , srv1795980542:86 , srv1894726413:86 , srv2034108042:86 , srv2114091763:86 , srv240242219:86 , srv366595783:86 , srv655949381:86 , srv8403639:86 , srv953291038:86 , srv1471229510:87 , srv1519784983:87 , srv1538198283:87 , srv1832965156:87 , srv1952371182:87 , srv414508150:87 , srv680007282:87 , srv705568401:87 , srv756555871:87 , srv788335463:87 , srv80210886:87 , srv844945477:87 , srv1618989914:88 , srv1796912734:88 , srv187832191:88 , srv330215771:88 , srv511570336:88 , srv696437562:88 , srv803495285:88 , srv846765651:88 , srv855938919:88 , srv1290738472:89 , srv1566582908:89 , srv1647632491:89 , srv1650431426:89 , srv1868473963:89 , srv2119831386:89 , srv255378570:89 , srv82642247:89 , srv1489501786:90 , srv446753645:90 , srv636703763:90 , srv915688649:90 , srv1165792890:91 , srv1530046590:91 , srv1991450488:91 , srv440330932:91 , srv556154365:91 , srv2128080621:92 , srv1909685667:481 } 2023-03-14 20:54:19,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:19,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33953718=60, srv1432411076=15, srv523207130=75, srv1252518198=4, srv1554849321=25, srv443665382=67, srv1720222672=32, srv953291038=98, srv1650431426=30, srv927190707=96, srv705568401=83, srv440330932=66, srv2119831386=54, srv414508150=63, srv1528466002=21, srv42680951=65, srv1988510570=48, srv929795704=97, srv1519784983=20, srv1280673857=7, srv2078107980=52, srv855938919=94, srv187832191=44, srv721835014=84, srv680007282=81, srv80210886=88, srv330215771=59, srv1538198283=24, srv1722042277=33, srv619910803=77, srv2128080621=55, srv1795980542=36, srv1299990919=9, srv1641596955=28, srv1290738472=8, srv42081658=64, srv1083060887=1, srv1275856101=5, srv366595783=62, srv4597393=70, srv240242219=57, srv1306781648=10, srv1842800908=41, srv636703763=78, srv1489501786=17, srv1656369887=31, srv1494696149=18, srv1165792890=2, srv1868473963=42, srv444913462=68, srv1875179654=43, srv1803007052=38, srv1747927134=34, srv1566582908=26, srv756555871=85, srv499240490=71, srv1279630480=6, srv217759238=56, srv515292452=74, srv1530046590=22, srv2114091763=53, srv79864763=87, srv1501977562=19, srv346142494=61, srv655949381=79, srv996052487=99, srv2034108042=50, srv1647632491=29, srv446753645=69, srv82642247=90, srv1380213936=11, srv1832965156=39, srv803495285=89, srv844945477=92, srv846765651=93, srv1397496143=13, srv1952371182=47, srv696437562=82, srv556154365=76, srv1471229510=16, srv2068058947=51, srv1533162100=23, srv788335463=86, srv1169252771=3, srv672257538=80, srv1796912734=37, srv915688649=95, srv1991450488=49, srv1384888874=12, srv511570336=73, srv1763241379=35, srv1414665857=14, srv1618989914=27, srv1894726413=45, srv1909685667=46, srv8403639=91, srv1009740286=0, srv1839252278=40, srv255378570=58, srv502433346=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:19,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-14 20:54:19,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-14 20:54:19,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-14 20:54:19,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-14 20:54:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-14 20:54:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-14 20:54:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-14 20:54:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-14 20:54:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-14 20:54:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-14 20:54:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-14 20:54:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-14 20:54:19,354 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04396997013984675, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06005609050491054, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04396997013984675, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:19,354 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-14 20:54:19,354 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17867682159772597, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04396997013984675, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06005609050491054, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04396997013984675, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-14 20:54:19,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 390 ms to try 5000 different iterations. Found a solution that moves 2612 regions; Going from a computed imbalance of 0.17867682159772597 to a new imbalance of 0.014509498938473732. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429656319143873); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03661612678507574, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2902222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429656319143873); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:19,651 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv440330932:78 , srv1489501786:79 , srv2128080621:79 , srv1530046590:80 , srv1165792890:81 , srv1991450488:81 , srv915688649:81 , srv1566582908:82 , srv1832965156:82 , srv1868473963:82 , srv187832191:82 , srv2119831386:82 , srv446753645:82 , srv556154365:82 , srv1471229510:83 , srv1952371182:83 , srv2034108042:83 , srv255378570:83 , srv414508150:83 , srv636703763:83 , srv803495285:83 , srv1275856101:84 , srv1279630480:84 , srv1501977562:84 , srv1618989914:84 , srv1650431426:84 , srv1722042277:84 , srv2078107980:84 , srv240242219:84 , srv1290738472:85 , srv1641596955:85 , srv1647632491:85 , srv2114091763:85 , srv330215771:85 , srv511570336:85 , srv655949381:85 , srv696437562:85 , srv705568401:85 , srv788335463:85 , srv82642247:85 , srv855938919:85 , srv1009740286:86 , srv1519784983:86 , srv1538198283:86 , srv1656369887:86 , srv1720222672:86 , srv444913462:86 , srv4597393:86 , srv515292452:86 , srv680007282:86 , srv756555871:86 , srv80210886:86 , srv8403639:86 , srv844945477:86 , srv846765651:86 , srv927190707:86 , srv1083060887:87 , srv1252518198:87 , srv1306781648:87 , srv1380213936:87 , srv1494696149:87 , srv1795980542:87 , srv1803007052:87 , srv217759238:87 , srv346142494:87 , srv366595783:87 , srv42081658:87 , srv42680951:87 , srv443665382:87 , srv502433346:87 , srv953291038:87 , srv996052487:87 , srv1169252771:88 , srv1299990919:88 , srv1414665857:88 , srv1432411076:88 , srv1528466002:88 , srv1747927134:88 , srv1796912734:88 , srv1842800908:88 , srv1894726413:88 , srv33953718:88 , srv619910803:88 , srv672257538:88 , srv1280673857:89 , srv1554849321:89 , srv1875179654:89 , srv523207130:89 , srv721835014:89 , srv79864763:89 , srv1384888874:90 , srv1397496143:90 , srv1763241379:90 , srv1839252278:90 , srv2068058947:90 , srv929795704:90 , srv1533162100:91 , srv1988510570:91 , srv499240490:91 , srv1909685667:509 } 2023-03-14 20:54:19,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:19,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33953718=60, srv1432411076=15, srv523207130=75, srv1252518198=4, srv1554849321=25, srv443665382=67, srv1720222672=32, srv953291038=98, srv1650431426=30, srv927190707=96, srv705568401=83, srv440330932=66, srv2119831386=54, srv414508150=63, srv1528466002=21, srv42680951=65, srv1988510570=48, srv929795704=97, srv1519784983=20, srv1280673857=7, srv2078107980=52, srv855938919=94, srv187832191=44, srv721835014=84, srv680007282=81, srv80210886=88, srv330215771=59, srv1538198283=24, srv1722042277=33, srv619910803=77, srv2128080621=55, srv1795980542=36, srv1299990919=9, srv1641596955=28, srv1290738472=8, srv42081658=64, srv1083060887=1, srv1275856101=5, srv366595783=62, srv4597393=70, srv240242219=57, srv1306781648=10, srv1842800908=41, srv636703763=78, srv1489501786=17, srv1656369887=31, srv1494696149=18, srv1165792890=2, srv1868473963=42, srv444913462=68, srv1875179654=43, srv1803007052=38, srv1747927134=34, srv1566582908=26, srv756555871=85, srv499240490=71, srv1279630480=6, srv217759238=56, srv515292452=74, srv1530046590=22, srv2114091763=53, srv79864763=87, srv1501977562=19, srv346142494=61, srv655949381=79, srv996052487=99, srv2034108042=50, srv1647632491=29, srv446753645=69, srv82642247=90, srv1380213936=11, srv1832965156=39, srv803495285=89, srv844945477=92, srv846765651=93, srv1397496143=13, srv1952371182=47, srv696437562=82, srv556154365=76, srv1471229510=16, srv2068058947=51, srv1533162100=23, srv788335463=86, srv1169252771=3, srv672257538=80, srv1796912734=37, srv915688649=95, srv1991450488=49, srv1384888874=12, srv511570336=73, srv1763241379=35, srv1414665857=14, srv1618989914=27, srv1894726413=45, srv1909685667=46, srv8403639=91, srv1009740286=0, srv1839252278=40, srv255378570=58, srv502433346=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-14 20:54:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-14 20:54:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-14 20:54:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-14 20:54:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-14 20:54:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-14 20:54:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-14 20:54:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-14 20:54:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-14 20:54:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-14 20:54:19,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-14 20:54:19,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-14 20:54:19,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-14 20:54:19,857 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429656319143873); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03661612678507574, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429656319143873); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:19,857 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-14 20:54:19,857 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014491209323561953, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429656319143873); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03661612678507574, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429656319143873); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-14 20:54:20,101 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 381 ms to try 5000 different iterations. Found a solution that moves 1037 regions; Going from a computed imbalance of 0.014491209323561953 to a new imbalance of 6.442776583693903E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007454396416772877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011872876973668779); MoveCostFunction : (multiplier=7.0, imbalance=0.11522222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007454396416772877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:20,104 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1528466002:83 , srv1796912734:83 , srv721835014:83 , srv82642247:83 , srv1280673857:84 , srv1290738472:84 , srv1380213936:84 , srv1397496143:84 , srv1432411076:84 , srv1519784983:84 , srv1538198283:84 , srv1647632491:84 , srv1795980542:84 , srv1842800908:84 , srv255378570:84 , srv366595783:84 , srv42680951:84 , srv511570336:84 , srv556154365:84 , srv696437562:84 , srv756555871:84 , srv846765651:84 , srv855938919:84 , srv953291038:84 , srv1165792890:85 , srv1169252771:85 , srv1306781648:85 , srv1494696149:85 , srv1530046590:85 , srv1533162100:85 , srv1566582908:85 , srv1618989914:85 , srv1641596955:85 , srv1650431426:85 , srv1656369887:85 , srv1747927134:85 , srv1763241379:85 , srv1839252278:85 , srv1875179654:85 , srv1894726413:85 , srv1988510570:85 , srv1991450488:85 , srv2119831386:85 , srv217759238:85 , srv330215771:85 , srv42081658:85 , srv499240490:85 , srv523207130:85 , srv636703763:85 , srv672257538:85 , srv680007282:85 , srv705568401:85 , srv80210886:85 , srv844945477:85 , srv996052487:85 , srv1009740286:86 , srv1252518198:86 , srv1299990919:86 , srv1384888874:86 , srv1501977562:86 , srv1554849321:86 , srv1720222672:86 , srv1803007052:86 , srv1868473963:86 , srv2068058947:86 , srv2128080621:86 , srv443665382:86 , srv446753645:86 , srv4597393:86 , srv515292452:86 , srv788335463:86 , srv79864763:86 , srv8403639:86 , srv915688649:86 , srv1083060887:87 , srv1279630480:87 , srv1414665857:87 , srv1471229510:87 , srv187832191:87 , srv2114091763:87 , srv240242219:87 , srv33953718:87 , srv346142494:87 , srv414508150:87 , srv444913462:87 , srv502433346:87 , srv619910803:87 , srv655949381:87 , srv803495285:87 , srv927190707:87 , srv1275856101:88 , srv1722042277:88 , srv1832965156:88 , srv1952371182:88 , srv2034108042:88 , srv2078107980:88 , srv440330932:88 , srv929795704:88 , srv1489501786:89 , srv1909685667:534 } 2023-03-14 20:54:20,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:20,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33953718=60, srv1432411076=15, srv523207130=75, srv1252518198=4, srv1554849321=25, srv443665382=67, srv1720222672=32, srv953291038=98, srv1650431426=30, srv927190707=96, srv705568401=83, srv440330932=66, srv2119831386=54, srv414508150=63, srv1528466002=21, srv42680951=65, srv1988510570=48, srv929795704=97, srv1519784983=20, srv1280673857=7, srv2078107980=52, srv855938919=94, srv187832191=44, srv721835014=84, srv680007282=81, srv80210886=88, srv330215771=59, srv1538198283=24, srv1722042277=33, srv619910803=77, srv2128080621=55, srv1795980542=36, srv1299990919=9, srv1641596955=28, srv1290738472=8, srv42081658=64, srv1083060887=1, srv1275856101=5, srv366595783=62, srv4597393=70, srv240242219=57, srv1306781648=10, srv1842800908=41, srv636703763=78, srv1489501786=17, srv1656369887=31, srv1494696149=18, srv1165792890=2, srv1868473963=42, srv444913462=68, srv1875179654=43, srv1803007052=38, srv1747927134=34, srv1566582908=26, srv756555871=85, srv499240490=71, srv1279630480=6, srv217759238=56, srv515292452=74, srv1530046590=22, srv2114091763=53, srv79864763=87, srv1501977562=19, srv346142494=61, srv655949381=79, srv996052487=99, srv2034108042=50, srv1647632491=29, srv446753645=69, srv82642247=90, srv1380213936=11, srv1832965156=39, srv803495285=89, srv844945477=92, srv846765651=93, srv1397496143=13, srv1952371182=47, srv696437562=82, srv556154365=76, srv1471229510=16, srv2068058947=51, srv1533162100=23, srv788335463=86, srv1169252771=3, srv672257538=80, srv1796912734=37, srv915688649=95, srv1991450488=49, srv1384888874=12, srv511570336=73, srv1763241379=35, srv1414665857=14, srv1618989914=27, srv1894726413=45, srv1909685667=46, srv8403639=91, srv1009740286=0, srv1839252278=40, srv255378570=58, srv502433346=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:20,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:20,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-14 20:54:20,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-14 20:54:20,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-14 20:54:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-14 20:54:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-14 20:54:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-14 20:54:20,233 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007454396416772877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011872876973668779); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007454396416772877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:20,234 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-14 20:54:20,234 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.370164291720268E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007454396416772877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011872876973668779); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007454396416772877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-14 20:54:20,613 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 477 ms to try 5000 different iterations. Found a solution that moves 292 regions; Going from a computed imbalance of 6.370164291720268E-4 to a new imbalance of 4.312897899147073E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03244444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:20,614 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1414665857:83 , srv1494696149:83 , srv1894726413:83 , srv1988510570:83 , srv2068058947:83 , srv1009740286:84 , srv1279630480:84 , srv1280673857:84 , srv1290738472:84 , srv1384888874:84 , srv1397496143:84 , srv1489501786:84 , srv1554849321:84 , srv1647632491:84 , srv1650431426:84 , srv1796912734:84 , srv1868473963:84 , srv1875179654:84 , srv1952371182:84 , srv2128080621:84 , srv33953718:84 , srv444913462:84 , srv446753645:84 , srv499240490:84 , srv636703763:84 , srv655949381:84 , srv680007282:84 , srv788335463:84 , srv79864763:84 , srv80210886:84 , srv803495285:84 , srv8403639:84 , srv844945477:84 , srv846765651:84 , srv929795704:84 , srv953291038:84 , srv1165792890:85 , srv1169252771:85 , srv1252518198:85 , srv1275856101:85 , srv1299990919:85 , srv1306781648:85 , srv1380213936:85 , srv1432411076:85 , srv1471229510:85 , srv1519784983:85 , srv1528466002:85 , srv1533162100:85 , srv1538198283:85 , srv1618989914:85 , srv1656369887:85 , srv1720222672:85 , srv1722042277:85 , srv1747927134:85 , srv1763241379:85 , srv1795980542:85 , srv1803007052:85 , srv1832965156:85 , srv1839252278:85 , srv1842800908:85 , srv187832191:85 , srv1991450488:85 , srv2034108042:85 , srv2078107980:85 , srv2114091763:85 , srv240242219:85 , srv255378570:85 , srv330215771:85 , srv346142494:85 , srv366595783:85 , srv414508150:85 , srv42081658:85 , srv440330932:85 , srv443665382:85 , srv4597393:85 , srv502433346:85 , srv511570336:85 , srv523207130:85 , srv556154365:85 , srv619910803:85 , srv672257538:85 , srv696437562:85 , srv705568401:85 , srv721835014:85 , srv756555871:85 , srv82642247:85 , srv855938919:85 , srv915688649:85 , srv927190707:85 , srv996052487:85 , srv1083060887:86 , srv1501977562:86 , srv1530046590:86 , srv1566582908:86 , srv1641596955:86 , srv2119831386:86 , srv217759238:86 , srv42680951:86 , srv515292452:86 , srv1909685667:617 } 2023-03-14 20:54:20,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:20,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33953718=60, srv1432411076=15, srv523207130=75, srv1252518198=4, srv1554849321=25, srv443665382=67, srv1720222672=32, srv953291038=98, srv1650431426=30, srv927190707=96, srv705568401=83, srv440330932=66, srv2119831386=54, srv414508150=63, srv1528466002=21, srv42680951=65, srv1988510570=48, srv929795704=97, srv1519784983=20, srv1280673857=7, srv2078107980=52, srv855938919=94, srv187832191=44, srv721835014=84, srv680007282=81, srv80210886=88, srv330215771=59, srv1538198283=24, srv1722042277=33, srv619910803=77, srv2128080621=55, srv1795980542=36, srv1299990919=9, srv1641596955=28, srv1290738472=8, srv42081658=64, srv1083060887=1, srv1275856101=5, srv366595783=62, srv4597393=70, srv240242219=57, srv1306781648=10, srv1842800908=41, srv636703763=78, srv1489501786=17, srv1656369887=31, srv1494696149=18, srv1165792890=2, srv1868473963=42, srv444913462=68, srv1875179654=43, srv1803007052=38, srv1747927134=34, srv1566582908=26, srv756555871=85, srv499240490=71, srv1279630480=6, srv217759238=56, srv515292452=74, srv1530046590=22, srv2114091763=53, srv79864763=87, srv1501977562=19, srv346142494=61, srv655949381=79, srv996052487=99, srv2034108042=50, srv1647632491=29, srv446753645=69, srv82642247=90, srv1380213936=11, srv1832965156=39, srv803495285=89, srv844945477=92, srv846765651=93, srv1397496143=13, srv1952371182=47, srv696437562=82, srv556154365=76, srv1471229510=16, srv2068058947=51, srv1533162100=23, srv788335463=86, srv1169252771=3, srv672257538=80, srv1796912734=37, srv915688649=95, srv1991450488=49, srv1384888874=12, srv511570336=73, srv1763241379=35, srv1414665857=14, srv1618989914=27, srv1894726413=45, srv1909685667=46, srv8403639=91, srv1009740286=0, srv1839252278=40, srv255378570=58, srv502433346=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:20,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:20,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-14 20:54:20,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-14 20:54:20,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-14 20:54:20,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-14 20:54:20,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-14 20:54:20,740 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:20,740 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-14 20:54:20,741 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.108435129076662E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-14 20:54:20,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 320 ms to try 5000 different iterations. Found a solution that moves 33 regions; Going from a computed imbalance of 4.108435129076662E-5 to a new imbalance of 7.618828439090763E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0036666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:20,965 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1083060887:83 , srv1384888874:83 , srv1533162100:83 , srv1796912734:83 , srv2114091763:83 , srv330215771:83 , srv346142494:83 , srv515292452:83 , srv929795704:83 , srv1009740286:84 , srv1165792890:84 , srv1169252771:84 , srv1252518198:84 , srv1275856101:84 , srv1280673857:84 , srv1290738472:84 , srv1299990919:84 , srv1380213936:84 , srv1397496143:84 , srv1414665857:84 , srv1432411076:84 , srv1471229510:84 , srv1489501786:84 , srv1501977562:84 , srv1519784983:84 , srv1528466002:84 , srv1530046590:84 , srv1538198283:84 , srv1554849321:84 , srv1566582908:84 , srv1618989914:84 , srv1641596955:84 , srv1647632491:84 , srv1650431426:84 , srv1656369887:84 , srv1720222672:84 , srv1722042277:84 , srv1747927134:84 , srv1763241379:84 , srv1795980542:84 , srv1832965156:84 , srv1839252278:84 , srv1842800908:84 , srv1875179654:84 , srv187832191:84 , srv1894726413:84 , srv1952371182:84 , srv1988510570:84 , srv1991450488:84 , srv2034108042:84 , srv2068058947:84 , srv2078107980:84 , srv2119831386:84 , srv217759238:84 , srv240242219:84 , srv33953718:84 , srv366595783:84 , srv414508150:84 , srv42081658:84 , srv42680951:84 , srv440330932:84 , srv443665382:84 , srv444913462:84 , srv446753645:84 , srv4597393:84 , srv499240490:84 , srv502433346:84 , srv511570336:84 , srv523207130:84 , srv556154365:84 , srv619910803:84 , srv636703763:84 , srv655949381:84 , srv672257538:84 , srv680007282:84 , srv696437562:84 , srv705568401:84 , srv721835014:84 , srv756555871:84 , srv788335463:84 , srv79864763:84 , srv80210886:84 , srv803495285:84 , srv82642247:84 , srv8403639:84 , srv844945477:84 , srv846765651:84 , srv855938919:84 , srv915688649:84 , srv927190707:84 , srv996052487:84 , srv1279630480:85 , srv1306781648:85 , srv1494696149:85 , srv1803007052:85 , srv1868473963:85 , srv2128080621:85 , srv255378570:85 , srv953291038:85 , srv1909685667:685 } 2023-03-14 20:54:20,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv33953718=60, srv1432411076=15, srv523207130=75, srv1252518198=4, srv1554849321=25, srv443665382=67, srv1720222672=32, srv953291038=98, srv1650431426=30, srv927190707=96, srv705568401=83, srv440330932=66, srv2119831386=54, srv414508150=63, srv1528466002=21, srv42680951=65, srv1988510570=48, srv929795704=97, srv1519784983=20, srv1280673857=7, srv2078107980=52, srv855938919=94, srv187832191=44, srv721835014=84, srv680007282=81, srv80210886=88, srv330215771=59, srv1538198283=24, srv1722042277=33, srv619910803=77, srv2128080621=55, srv1795980542=36, srv1299990919=9, srv1641596955=28, srv1290738472=8, srv42081658=64, srv1083060887=1, srv1275856101=5, srv366595783=62, srv4597393=70, srv240242219=57, srv1306781648=10, srv1842800908=41, srv636703763=78, srv1489501786=17, srv1656369887=31, srv1494696149=18, srv1165792890=2, srv1868473963=42, srv444913462=68, srv1875179654=43, srv1803007052=38, srv1747927134=34, srv1566582908=26, srv756555871=85, srv499240490=71, srv1279630480=6, srv217759238=56, srv515292452=74, srv1530046590=22, srv2114091763=53, srv79864763=87, srv1501977562=19, srv346142494=61, srv655949381=79, srv996052487=99, srv2034108042=50, srv1647632491=29, srv446753645=69, srv82642247=90, srv1380213936=11, srv1832965156=39, srv803495285=89, srv844945477=92, srv846765651=93, srv1397496143=13, srv1952371182=47, srv696437562=82, srv556154365=76, srv1471229510=16, srv2068058947=51, srv1533162100=23, srv788335463=86, srv1169252771=3, srv672257538=80, srv1796912734=37, srv915688649=95, srv1991450488=49, srv1384888874=12, srv511570336=73, srv1763241379=35, srv1414665857=14, srv1618989914=27, srv1894726413=45, srv1909685667=46, srv8403639=91, srv1009740286=0, srv1839252278=40, srv255378570=58, srv502433346=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:20,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:20,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-14 20:54:20,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-14 20:54:20,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-14 20:54:20,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-14 20:54:20,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-14 20:54:20,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-14 20:54:20,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-14 20:54:21,094 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:21,095 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1083060887:83 , srv1384888874:83 , srv1533162100:83 , srv1796912734:83 , srv2114091763:83 , srv330215771:83 , srv346142494:83 , srv515292452:83 , srv929795704:83 , srv1009740286:84 , srv1165792890:84 , srv1169252771:84 , srv1252518198:84 , srv1275856101:84 , srv1280673857:84 , srv1290738472:84 , srv1299990919:84 , srv1380213936:84 , srv1397496143:84 , srv1414665857:84 , srv1432411076:84 , srv1471229510:84 , srv1489501786:84 , srv1501977562:84 , srv1519784983:84 , srv1528466002:84 , srv1530046590:84 , srv1538198283:84 , srv1554849321:84 , srv1566582908:84 , srv1618989914:84 , srv1641596955:84 , srv1647632491:84 , srv1650431426:84 , srv1656369887:84 , srv1720222672:84 , srv1722042277:84 , srv1747927134:84 , srv1763241379:84 , srv1795980542:84 , srv1832965156:84 , srv1839252278:84 , srv1842800908:84 , srv1875179654:84 , srv187832191:84 , srv1894726413:84 , srv1952371182:84 , srv1988510570:84 , srv1991450488:84 , srv2034108042:84 , srv2068058947:84 , srv2078107980:84 , srv2119831386:84 , srv217759238:84 , srv240242219:84 , srv33953718:84 , srv366595783:84 , srv414508150:84 , srv42081658:84 , srv42680951:84 , srv440330932:84 , srv443665382:84 , srv444913462:84 , srv446753645:84 , srv4597393:84 , srv499240490:84 , srv502433346:84 , srv511570336:84 , srv523207130:84 , srv556154365:84 , srv619910803:84 , srv636703763:84 , srv655949381:84 , srv672257538:84 , srv680007282:84 , srv696437562:84 , srv705568401:84 , srv721835014:84 , srv756555871:84 , srv788335463:84 , srv79864763:84 , srv80210886:84 , srv803495285:84 , srv82642247:84 , srv8403639:84 , srv844945477:84 , srv846765651:84 , srv855938919:84 , srv915688649:84 , srv927190707:84 , srv996052487:84 , srv1279630480:85 , srv1306781648:85 , srv1494696149:85 , srv1803007052:85 , srv1868473963:85 , srv2128080621:85 , srv255378570:85 , srv953291038:85 , srv1909685667:685 } 2023-03-14 20:54:21,118 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=169 (was 169), AvailableMemoryMB=3206 (was 3742) 2023-03-14 20:54:21,125 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=169, AvailableMemoryMB=3205 2023-03-14 20:54:21,125 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-14 20:54:21,125 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-14 20:54:21,133 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1064497899:84 , srv1117597623:84 , srv1149089167:84 , srv1247775890:84 , srv1286650679:84 , srv1318663955:84 , srv1390684376:84 , srv1445754976:84 , srv1487539272:84 , srv1531238027:84 , srv153496541:84 , srv1545712573:84 , srv1579432519:84 , srv1599285894:84 , srv1753642945:84 , srv1839561307:84 , srv1947085638:84 , srv1954340221:84 , srv2038668853:84 , srv2144965930:84 , srv241488885:84 , srv252502420:84 , srv397755535:84 , srv412316700:84 , srv437264993:84 , srv681217088:84 , srv784174736:84 , srv82869007:84 , srv888609983:84 , srv863373732:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-14 20:54:21,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:54:21,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1445754976=7, srv1487539272=8, srv397755535=22, srv1579432519=12, srv1839561307=15, srv1947085638=16, srv1117597623=1, srv1247775890=3, srv252502420=21, srv888609983=29, srv241488885=20, srv681217088=25, srv412316700=23, srv82869007=27, srv1064497899=0, srv784174736=26, srv1149089167=2, srv1390684376=6, srv1954340221=17, srv1545712573=11, srv1531238027=9, srv153496541=10, srv2038668853=18, srv437264993=24, srv1318663955=5, srv1286650679=4, srv1753642945=14, srv1599285894=13, srv2144965930=19, srv863373732=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:54:21,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:54:21,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:54:21,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:54:21,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:54:21,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:54:21,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:54:21,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:54:21,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:54:21,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-14 20:54:21,159 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0762455958869974, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:54:21,160 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912433290647272, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0762455958869974, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-14 20:56:21,147 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6189194 different iterations. Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912433290647272 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:56:21,154 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1064497899:90 , srv1117597623:90 , srv1149089167:90 , srv1247775890:90 , srv1286650679:90 , srv1318663955:90 , srv1390684376:90 , srv1445754976:90 , srv1487539272:90 , srv1531238027:90 , srv153496541:90 , srv1545712573:90 , srv1579432519:90 , srv1599285894:90 , srv1753642945:90 , srv1839561307:90 , srv1947085638:90 , srv1954340221:90 , srv2038668853:90 , srv2144965930:90 , srv241488885:90 , srv252502420:90 , srv397755535:90 , srv412316700:90 , srv437264993:90 , srv681217088:90 , srv784174736:90 , srv82869007:90 , srv863373732:90 , srv888609983:90 } 2023-03-14 20:56:21,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-14 20:56:21,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1445754976=7, srv1487539272=8, srv397755535=22, srv1579432519=12, srv1839561307=15, srv1947085638=16, srv1117597623=1, srv1247775890=3, srv252502420=21, srv888609983=29, srv241488885=20, srv681217088=25, srv412316700=23, srv82869007=27, srv1064497899=0, srv784174736=26, srv1149089167=2, srv1390684376=6, srv1954340221=17, srv1545712573=11, srv1531238027=9, srv153496541=10, srv2038668853=18, srv437264993=24, srv1318663955=5, srv1286650679=4, srv1753642945=14, srv1599285894=13, srv2144965930=19, srv863373732=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-14 20:56:21,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-14 20:56:21,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-14 20:56:21,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-14 20:56:21,176 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-14 20:56:21,176 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1064497899:90 , srv1117597623:90 , srv1149089167:90 , srv1247775890:90 , srv1286650679:90 , srv1318663955:90 , srv1390684376:90 , srv1445754976:90 , srv1487539272:90 , srv1531238027:90 , srv153496541:90 , srv1545712573:90 , srv1579432519:90 , srv1599285894:90 , srv1753642945:90 , srv1839561307:90 , srv1947085638:90 , srv1954340221:90 , srv2038668853:90 , srv2144965930:90 , srv241488885:90 , srv252502420:90 , srv397755535:90 , srv412316700:90 , srv437264993:90 , srv681217088:90 , srv784174736:90 , srv82869007:90 , srv863373732:90 , srv888609983:90 } 2023-03-14 20:56:21,188 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=160 (was 241), ProcessCount=169 (was 169), AvailableMemoryMB=2872 (was 3205)