2023-09-23 06:53:16,421 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-23 06:53:16,754 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-23 06:53:16,871 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-23 06:53:16,941 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 06:53:16,949 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-23 06:53:16,966 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=173, AvailableMemoryMB=3099 2023-09-23 06:53:16,968 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 06:53:16,968 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-23 06:53:17,109 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002113764:84 , srv1016209812:84 , srv1030761298:84 , srv1032180724:84 , srv1038802280:84 , srv1050859309:84 , srv1051204705:84 , srv1055875693:84 , srv1057183946:84 , srv1114486717:84 , srv1116497917:84 , srv1147194960:84 , srv11532620:84 , srv1153883546:84 , srv1177096377:84 , srv119212329:84 , srv1229021248:84 , srv1261427588:84 , srv1273487486:84 , srv1281730714:84 , srv1370366172:84 , srv1376444412:84 , srv1448282240:84 , srv1460952760:84 , srv1493150358:84 , srv149360107:84 , srv1497514520:84 , srv1498590543:84 , srv1512798883:84 , srv1516558149:84 , srv1527288398:84 , srv1529220363:84 , srv156195638:84 , srv1565581232:84 , srv1586408404:84 , srv1608979671:84 , srv1618646630:84 , srv1619074424:84 , srv1621794055:84 , srv1632594360:84 , srv1675814496:84 , srv1728799862:84 , srv1731202927:84 , srv1773616809:84 , srv1799012722:84 , srv1825578919:84 , srv1837478273:84 , srv1838793445:84 , srv185961857:84 , srv1869006553:84 , srv1869857427:84 , srv1904442761:84 , srv1906386455:84 , srv1960679316:84 , srv1993560338:84 , srv2040576814:84 , srv2092063459:84 , srv2107256971:84 , srv2111532206:84 , srv256134833:84 , srv262709016:84 , srv272702626:84 , srv313701950:84 , srv322782311:84 , srv332698058:84 , srv403627530:84 , srv412838327:84 , srv419520826:84 , srv421774562:84 , srv46557575:84 , srv480351088:84 , srv492607409:84 , srv503103567:84 , srv530674749:84 , srv530881598:84 , srv540866248:84 , srv607928680:84 , srv612723731:84 , srv617605763:84 , srv617843363:84 , srv642859496:84 , srv647282321:84 , srv682214246:84 , srv722559490:84 , srv732893009:84 , srv737023316:84 , srv758369287:84 , srv759370070:84 , srv767644587:84 , srv791379749:84 , srv806681177:84 , srv837920766:84 , srv851017121:84 , srv853728802:84 , srv857245914:84 , srv889243276:84 , srv921521792:84 , srv960694513:84 , srv961033361:84 , srv86295221:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-23 06:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906386455=52, srv2107256971=57, srv1960679316=53, srv1493150358=24, srv1229021248=16, srv851017121=92, srv480351088=70, srv1527288398=30, srv1448282240=22, srv1153883546=13, srv961033361=99, srv149360107=25, srv1516558149=29, srv185961857=48, srv1460952760=23, srv642859496=80, srv403627530=65, srv2111532206=58, srv612723731=77, srv791379749=89, srv1838793445=47, srv853728802=93, srv1281730714=19, srv732893009=84, srv1114486717=9, srv759370070=87, srv503103567=72, srv2092063459=56, srv1497514520=26, srv617605763=78, srv262709016=60, srv1032180724=3, srv156195638=32, srv607928680=76, srv647282321=81, srv11532620=12, srv1825578919=45, srv737023316=85, srv332698058=64, srv1147194960=11, srv119212329=15, srv1261427588=17, srv1621794055=38, srv272702626=61, srv767644587=88, srv1512798883=28, srv1565581232=33, srv889243276=96, srv1116497917=10, srv1030761298=2, srv1731202927=42, srv46557575=69, srv1618646630=36, srv1050859309=5, srv313701950=62, srv758369287=86, srv1016209812=1, srv1799012722=44, srv1837478273=46, srv86295221=95, srv1619074424=37, srv256134833=59, srv421774562=68, srv530881598=74, srv1675814496=40, srv412838327=66, srv857245914=94, srv1177096377=14, srv1370366172=20, srv682214246=82, srv1498590543=27, srv1529220363=31, srv1057183946=8, srv1273487486=18, srv419520826=67, srv1376444412=21, srv1869006553=49, srv492607409=71, srv530674749=73, srv1728799862=41, srv1586408404=34, srv837920766=91, srv322782311=63, srv960694513=98, srv1002113764=0, srv1055875693=7, srv617843363=79, srv1038802280=4, srv1608979671=35, srv722559490=83, srv1773616809=43, srv540866248=75, srv1993560338=54, srv921521792=97, srv2040576814=55, srv1632594360=39, srv1904442761=51, srv806681177=90, srv1051204705=6, srv1869857427=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 06:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 06:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 06:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 06:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 06:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 06:53:17,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 06:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 06:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 06:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 06:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 06:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 06:53:17,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 06:53:17,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 06:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 06:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 06:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 06:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 06:53:17,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 06:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 06:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 06:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 06:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 06:53:17,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 06:53:17,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:17,421 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 06:53:17,421 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 06:53:17,774 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 524 ms to try 5000 different iterations. Found a solution that moves 4669 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1809279974236793. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959074841449644, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05592001663431077, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5187777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959074841449644, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:17,791 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1460952760:83 , srv1993560338:83 , srv612723731:83 , srv1032180724:84 , srv1050859309:84 , srv1497514520:84 , srv1586408404:84 , srv1619074424:84 , srv1837478273:84 , srv185961857:84 , srv332698058:84 , srv403627530:84 , srv421774562:84 , srv492607409:84 , srv530881598:84 , srv758369287:84 , srv853728802:84 , srv1051204705:85 , srv1114486717:85 , srv1147194960:85 , srv11532620:85 , srv1177096377:85 , srv1376444412:85 , srv1498590543:85 , srv1516558149:85 , srv1608979671:85 , srv1675814496:85 , srv1799012722:85 , srv1825578919:85 , srv1838793445:85 , srv2040576814:85 , srv2092063459:85 , srv2107256971:85 , srv262709016:85 , srv322782311:85 , srv412838327:85 , srv419520826:85 , srv46557575:85 , srv530674749:85 , srv607928680:85 , srv617843363:85 , srv682214246:85 , srv722559490:85 , srv737023316:85 , srv767644587:85 , srv791379749:85 , srv837920766:85 , srv1030761298:86 , srv1055875693:86 , srv1527288398:86 , srv1728799862:86 , srv1731202927:86 , srv1869006553:86 , srv1906386455:86 , srv2111532206:86 , srv256134833:86 , srv313701950:86 , srv480351088:86 , srv617605763:86 , srv647282321:86 , srv921521792:86 , srv1057183946:87 , srv1153883546:87 , srv1273487486:87 , srv149360107:87 , srv1512798883:87 , srv156195638:87 , srv1621794055:87 , srv1960679316:87 , srv732893009:87 , srv759370070:87 , srv851017121:87 , srv1229021248:88 , srv1493150358:88 , srv1773616809:88 , srv1869857427:88 , srv1904442761:88 , srv272702626:88 , srv642859496:88 , srv889243276:88 , srv961033361:88 , srv1016209812:89 , srv1448282240:89 , srv1529220363:89 , srv960694513:89 , srv1002113764:90 , srv1261427588:90 , srv503103567:90 , srv540866248:90 , srv857245914:90 , srv1038802280:91 , srv119212329:91 , srv1281730714:91 , srv806681177:91 , srv1116497917:92 , srv1370366172:92 , srv1565581232:92 , srv1618646630:92 , srv1632594360:92 , srv86295221:442 } 2023-09-23 06:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906386455=52, srv2107256971=57, srv1960679316=53, srv1493150358=24, srv1229021248=16, srv851017121=92, srv480351088=70, srv1527288398=30, srv1448282240=22, srv1153883546=13, srv961033361=99, srv149360107=25, srv1516558149=29, srv185961857=48, srv1460952760=23, srv642859496=80, srv403627530=65, srv2111532206=58, srv612723731=77, srv791379749=89, srv1838793445=47, srv853728802=93, srv1281730714=19, srv732893009=84, srv1114486717=9, srv759370070=87, srv503103567=72, srv2092063459=56, srv1497514520=26, srv617605763=78, srv262709016=60, srv1032180724=3, srv156195638=32, srv607928680=76, srv647282321=81, srv11532620=12, srv1825578919=45, srv737023316=85, srv332698058=64, srv1147194960=11, srv119212329=15, srv1261427588=17, srv1621794055=38, srv272702626=61, srv767644587=88, srv1512798883=28, srv1565581232=33, srv889243276=96, srv1116497917=10, srv1030761298=2, srv1731202927=42, srv46557575=69, srv1618646630=36, srv1050859309=5, srv313701950=62, srv758369287=86, srv1016209812=1, srv1799012722=44, srv1837478273=46, srv86295221=95, srv1619074424=37, srv256134833=59, srv421774562=68, srv530881598=74, srv1675814496=40, srv412838327=66, srv857245914=94, srv1177096377=14, srv1370366172=20, srv682214246=82, srv1498590543=27, srv1529220363=31, srv1057183946=8, srv1273487486=18, srv419520826=67, srv1376444412=21, srv1869006553=49, srv492607409=71, srv530674749=73, srv1728799862=41, srv1586408404=34, srv837920766=91, srv322782311=63, srv960694513=98, srv1002113764=0, srv1055875693=7, srv617843363=79, srv1038802280=4, srv1608979671=35, srv722559490=83, srv1773616809=43, srv540866248=75, srv1993560338=54, srv921521792=97, srv2040576814=55, srv1632594360=39, srv1904442761=51, srv806681177=90, srv1051204705=6, srv1869857427=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:17,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:17,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 06:53:17,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 06:53:17,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 06:53:17,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 06:53:17,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 06:53:17,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 06:53:17,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 06:53:17,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 06:53:17,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 06:53:17,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 06:53:17,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 06:53:17,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 06:53:18,009 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959074841449644, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05592001663431077, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959074841449644, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:18,009 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 06:53:18,009 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18089530438691698, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959074841449644, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05592001663431077, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959074841449644, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 06:53:18,370 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations. Found a solution that moves 2692 regions; Going from a computed imbalance of 0.18089530438691698 to a new imbalance of 0.015126695474107226. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02170957505533968); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03303910549004757, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2991111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02170957505533968); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:18,394 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116497917:79 , srv1370366172:80 , srv1281730714:81 , srv1565581232:81 , srv1632594360:81 , srv540866248:81 , srv1002113764:82 , srv1038802280:82 , srv119212329:82 , srv1261427588:82 , srv1448282240:82 , srv1529220363:82 , srv1618646630:82 , srv857245914:82 , srv1016209812:83 , srv806681177:83 , srv961033361:83 , srv1057183946:84 , srv1273487486:84 , srv1493150358:84 , srv149360107:84 , srv1512798883:84 , srv1608979671:84 , srv1904442761:84 , srv1906386455:84 , srv642859496:84 , srv851017121:84 , srv921521792:84 , srv960694513:84 , srv1030761298:85 , srv1051204705:85 , srv1055875693:85 , srv1177096377:85 , srv1229021248:85 , srv1728799862:85 , srv1773616809:85 , srv2107256971:85 , srv272702626:85 , srv412838327:85 , srv492607409:85 , srv503103567:85 , srv617605763:85 , srv732893009:85 , srv11532620:86 , srv1153883546:86 , srv1498590543:86 , srv1516558149:86 , srv1621794055:86 , srv1731202927:86 , srv1799012722:86 , srv1869857427:86 , srv1960679316:86 , srv256134833:86 , srv46557575:86 , srv530674749:86 , srv530881598:86 , srv647282321:86 , srv682214246:86 , srv722559490:86 , srv759370070:86 , srv889243276:86 , srv1114486717:87 , srv1147194960:87 , srv1497514520:87 , srv1527288398:87 , srv156195638:87 , srv1586408404:87 , srv1825578919:87 , srv2111532206:87 , srv262709016:87 , srv332698058:87 , srv419520826:87 , srv480351088:87 , srv617843363:87 , srv767644587:87 , srv853728802:87 , srv1376444412:88 , srv1460952760:88 , srv1619074424:88 , srv1837478273:88 , srv1838793445:88 , srv185961857:88 , srv1869006553:88 , srv2040576814:88 , srv2092063459:88 , srv313701950:88 , srv322782311:88 , srv403627530:88 , srv421774562:88 , srv737023316:88 , srv791379749:88 , srv837920766:88 , srv1032180724:89 , srv1050859309:89 , srv1675814496:89 , srv607928680:89 , srv612723731:89 , srv1993560338:90 , srv758369287:93 , srv86295221:525 } 2023-09-23 06:53:18,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:18,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906386455=52, srv2107256971=57, srv1960679316=53, srv1493150358=24, srv1229021248=16, srv851017121=92, srv480351088=70, srv1527288398=30, srv1448282240=22, srv1153883546=13, srv961033361=99, srv149360107=25, srv1516558149=29, srv185961857=48, srv1460952760=23, srv642859496=80, srv403627530=65, srv2111532206=58, srv612723731=77, srv791379749=89, srv1838793445=47, srv853728802=93, srv1281730714=19, srv732893009=84, srv1114486717=9, srv759370070=87, srv503103567=72, srv2092063459=56, srv1497514520=26, srv617605763=78, srv262709016=60, srv1032180724=3, srv156195638=32, srv607928680=76, srv647282321=81, srv11532620=12, srv1825578919=45, srv737023316=85, srv332698058=64, srv1147194960=11, srv119212329=15, srv1261427588=17, srv1621794055=38, srv272702626=61, srv767644587=88, srv1512798883=28, srv1565581232=33, srv889243276=96, srv1116497917=10, srv1030761298=2, srv1731202927=42, srv46557575=69, srv1618646630=36, srv1050859309=5, srv313701950=62, srv758369287=86, srv1016209812=1, srv1799012722=44, srv1837478273=46, srv86295221=95, srv1619074424=37, srv256134833=59, srv421774562=68, srv530881598=74, srv1675814496=40, srv412838327=66, srv857245914=94, srv1177096377=14, srv1370366172=20, srv682214246=82, srv1498590543=27, srv1529220363=31, srv1057183946=8, srv1273487486=18, srv419520826=67, srv1376444412=21, srv1869006553=49, srv492607409=71, srv530674749=73, srv1728799862=41, srv1586408404=34, srv837920766=91, srv322782311=63, srv960694513=98, srv1002113764=0, srv1055875693=7, srv617843363=79, srv1038802280=4, srv1608979671=35, srv722559490=83, srv1773616809=43, srv540866248=75, srv1993560338=54, srv921521792=97, srv2040576814=55, srv1632594360=39, srv1904442761=51, srv806681177=90, srv1051204705=6, srv1869857427=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:18,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:18,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:18,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:18,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 06:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 06:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 06:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 06:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 06:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 06:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 06:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 06:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 06:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 06:53:18,615 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02170957505533968); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03303910549004757, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02170957505533968); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:18,615 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 06:53:18,615 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015107845687222652, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02170957505533968); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03303910549004757, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02170957505533968); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 06:53:18,982 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations. Found a solution that moves 972 regions; Going from a computed imbalance of 0.015107845687222652 to a new imbalance of 6.497987633651001E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.108, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:18,986 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv758369287:81 , srv837920766:83 , srv1038802280:84 , srv1050859309:84 , srv1153883546:84 , srv119212329:84 , srv156195638:84 , srv1565581232:84 , srv1618646630:84 , srv1838793445:84 , srv1869006553:84 , srv1869857427:84 , srv2092063459:84 , srv272702626:84 , srv313701950:84 , srv503103567:84 , srv607928680:84 , srv791379749:84 , srv806681177:84 , srv889243276:84 , srv960694513:84 , srv1002113764:85 , srv1016209812:85 , srv1032180724:85 , srv1114486717:85 , srv1229021248:85 , srv1261427588:85 , srv1281730714:85 , srv1370366172:85 , srv1376444412:85 , srv1493150358:85 , srv1527288398:85 , srv1619074424:85 , srv1621794055:85 , srv1632594360:85 , srv1773616809:85 , srv1825578919:85 , srv1837478273:85 , srv185961857:85 , srv1904442761:85 , srv1960679316:85 , srv1993560338:85 , srv2040576814:85 , srv2111532206:85 , srv256134833:85 , srv262709016:85 , srv322782311:85 , srv419520826:85 , srv480351088:85 , srv612723731:85 , srv617843363:85 , srv642859496:85 , srv647282321:85 , srv737023316:85 , srv759370070:85 , srv767644587:85 , srv857245914:85 , srv1147194960:86 , srv11532620:86 , srv1273487486:86 , srv1448282240:86 , srv1497514520:86 , srv1512798883:86 , srv1516558149:86 , srv1586408404:86 , srv1675814496:86 , srv1728799862:86 , srv1731202927:86 , srv403627530:86 , srv412838327:86 , srv421774562:86 , srv46557575:86 , srv722559490:86 , srv732893009:86 , srv1030761298:87 , srv1055875693:87 , srv1057183946:87 , srv1116497917:87 , srv1177096377:87 , srv1460952760:87 , srv149360107:87 , srv1498590543:87 , srv1529220363:87 , srv1799012722:87 , srv1906386455:87 , srv2107256971:87 , srv332698058:87 , srv530674749:87 , srv530881598:87 , srv540866248:87 , srv617605763:87 , srv682214246:87 , srv851017121:87 , srv853728802:87 , srv921521792:87 , srv961033361:87 , srv1051204705:88 , srv1608979671:88 , srv492607409:88 , srv86295221:540 } 2023-09-23 06:53:18,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:18,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906386455=52, srv2107256971=57, srv1960679316=53, srv1493150358=24, srv1229021248=16, srv851017121=92, srv480351088=70, srv1527288398=30, srv1448282240=22, srv1153883546=13, srv961033361=99, srv149360107=25, srv1516558149=29, srv185961857=48, srv1460952760=23, srv642859496=80, srv403627530=65, srv2111532206=58, srv612723731=77, srv791379749=89, srv1838793445=47, srv853728802=93, srv1281730714=19, srv732893009=84, srv1114486717=9, srv759370070=87, srv503103567=72, srv2092063459=56, srv1497514520=26, srv617605763=78, srv262709016=60, srv1032180724=3, srv156195638=32, srv607928680=76, srv647282321=81, srv11532620=12, srv1825578919=45, srv737023316=85, srv332698058=64, srv1147194960=11, srv119212329=15, srv1261427588=17, srv1621794055=38, srv272702626=61, srv767644587=88, srv1512798883=28, srv1565581232=33, srv889243276=96, srv1116497917=10, srv1030761298=2, srv1731202927=42, srv46557575=69, srv1618646630=36, srv1050859309=5, srv313701950=62, srv758369287=86, srv1016209812=1, srv1799012722=44, srv1837478273=46, srv86295221=95, srv1619074424=37, srv256134833=59, srv421774562=68, srv530881598=74, srv1675814496=40, srv412838327=66, srv857245914=94, srv1177096377=14, srv1370366172=20, srv682214246=82, srv1498590543=27, srv1529220363=31, srv1057183946=8, srv1273487486=18, srv419520826=67, srv1376444412=21, srv1869006553=49, srv492607409=71, srv530674749=73, srv1728799862=41, srv1586408404=34, srv837920766=91, srv322782311=63, srv960694513=98, srv1002113764=0, srv1055875693=7, srv617843363=79, srv1038802280=4, srv1608979671=35, srv722559490=83, srv1773616809=43, srv540866248=75, srv1993560338=54, srv921521792=97, srv2040576814=55, srv1632594360=39, srv1904442761=51, srv806681177=90, srv1051204705=6, srv1869857427=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 06:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 06:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 06:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 06:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 06:53:19,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 06:53:19,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 06:53:19,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 06:53:19,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 06:53:19,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 06:53:19,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 06:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 06:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 06:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 06:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 06:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 06:53:19,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 06:53:19,198 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:19,198 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 06:53:19,204 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.42992673895633E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 06:53:19,649 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 593 ms to try 5000 different iterations. Found a solution that moves 233 regions; Going from a computed imbalance of 6.42992673895633E-4 to a new imbalance of 3.468012270112198E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:19,650 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1675814496:82 , srv1030761298:83 , srv1055875693:83 , srv1529220363:83 , srv2040576814:83 , srv503103567:83 , srv1032180724:84 , srv1050859309:84 , srv1051204705:84 , srv1057183946:84 , srv1116497917:84 , srv1147194960:84 , srv119212329:84 , srv1229021248:84 , srv1281730714:84 , srv1376444412:84 , srv1460952760:84 , srv149360107:84 , srv1498590543:84 , srv156195638:84 , srv1608979671:84 , srv1618646630:84 , srv1621794055:84 , srv1632594360:84 , srv1731202927:84 , srv1773616809:84 , srv1869006553:84 , srv1869857427:84 , srv1960679316:84 , srv1993560338:84 , srv2111532206:84 , srv313701950:84 , srv322782311:84 , srv332698058:84 , srv403627530:84 , srv421774562:84 , srv480351088:84 , srv530674749:84 , srv540866248:84 , srv607928680:84 , srv612723731:84 , srv617605763:84 , srv737023316:84 , srv758369287:84 , srv759370070:84 , srv806681177:84 , srv851017121:84 , srv853728802:84 , srv889243276:84 , srv961033361:84 , srv1002113764:85 , srv1016209812:85 , srv1038802280:85 , srv11532620:85 , srv1153883546:85 , srv1177096377:85 , srv1261427588:85 , srv1273487486:85 , srv1448282240:85 , srv1512798883:85 , srv1516558149:85 , srv1527288398:85 , srv1565581232:85 , srv1586408404:85 , srv1619074424:85 , srv1728799862:85 , srv1799012722:85 , srv1825578919:85 , srv1837478273:85 , srv1838793445:85 , srv185961857:85 , srv1904442761:85 , srv1906386455:85 , srv2092063459:85 , srv2107256971:85 , srv256134833:85 , srv262709016:85 , srv272702626:85 , srv419520826:85 , srv46557575:85 , srv492607409:85 , srv530881598:85 , srv617843363:85 , srv642859496:85 , srv647282321:85 , srv682214246:85 , srv722559490:85 , srv732893009:85 , srv767644587:85 , srv791379749:85 , srv857245914:85 , srv921521792:85 , srv960694513:85 , srv1114486717:86 , srv1370366172:86 , srv1493150358:86 , srv1497514520:86 , srv412838327:86 , srv837920766:86 , srv86295221:636 } 2023-09-23 06:53:19,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:19,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906386455=52, srv2107256971=57, srv1960679316=53, srv1493150358=24, srv1229021248=16, srv851017121=92, srv480351088=70, srv1527288398=30, srv1448282240=22, srv1153883546=13, srv961033361=99, srv149360107=25, srv1516558149=29, srv185961857=48, srv1460952760=23, srv642859496=80, srv403627530=65, srv2111532206=58, srv612723731=77, srv791379749=89, srv1838793445=47, srv853728802=93, srv1281730714=19, srv732893009=84, srv1114486717=9, srv759370070=87, srv503103567=72, srv2092063459=56, srv1497514520=26, srv617605763=78, srv262709016=60, srv1032180724=3, srv156195638=32, srv607928680=76, srv647282321=81, srv11532620=12, srv1825578919=45, srv737023316=85, srv332698058=64, srv1147194960=11, srv119212329=15, srv1261427588=17, srv1621794055=38, srv272702626=61, srv767644587=88, srv1512798883=28, srv1565581232=33, srv889243276=96, srv1116497917=10, srv1030761298=2, srv1731202927=42, srv46557575=69, srv1618646630=36, srv1050859309=5, srv313701950=62, srv758369287=86, srv1016209812=1, srv1799012722=44, srv1837478273=46, srv86295221=95, srv1619074424=37, srv256134833=59, srv421774562=68, srv530881598=74, srv1675814496=40, srv412838327=66, srv857245914=94, srv1177096377=14, srv1370366172=20, srv682214246=82, srv1498590543=27, srv1529220363=31, srv1057183946=8, srv1273487486=18, srv419520826=67, srv1376444412=21, srv1869006553=49, srv492607409=71, srv530674749=73, srv1728799862=41, srv1586408404=34, srv837920766=91, srv322782311=63, srv960694513=98, srv1002113764=0, srv1055875693=7, srv617843363=79, srv1038802280=4, srv1608979671=35, srv722559490=83, srv1773616809=43, srv540866248=75, srv1993560338=54, srv921521792=97, srv2040576814=55, srv1632594360=39, srv1904442761=51, srv806681177=90, srv1051204705=6, srv1869857427=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 06:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 06:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 06:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 06:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 06:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 06:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 06:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 06:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 06:53:19,820 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:19,820 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 06:53:19,820 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.3048621830354657E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 06:53:20,047 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 350 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 3.3048621830354657E-5 to a new imbalance of 7.234288738508196E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:20,048 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1114486717:83 , srv1370366172:83 , srv1493150358:83 , srv1497514520:83 , srv1527288398:83 , srv1799012722:83 , srv682214246:83 , srv732893009:83 , srv1002113764:84 , srv1016209812:84 , srv1032180724:84 , srv1038802280:84 , srv1051204705:84 , srv1057183946:84 , srv1116497917:84 , srv1147194960:84 , srv11532620:84 , srv1153883546:84 , srv1177096377:84 , srv1229021248:84 , srv1261427588:84 , srv1273487486:84 , srv1376444412:84 , srv1448282240:84 , srv1460952760:84 , srv149360107:84 , srv1498590543:84 , srv1512798883:84 , srv1516558149:84 , srv156195638:84 , srv1565581232:84 , srv1586408404:84 , srv1618646630:84 , srv1619074424:84 , srv1621794055:84 , srv1632594360:84 , srv1675814496:84 , srv1728799862:84 , srv1731202927:84 , srv1773616809:84 , srv1825578919:84 , srv1837478273:84 , srv1838793445:84 , srv185961857:84 , srv1869006553:84 , srv1869857427:84 , srv1904442761:84 , srv1906386455:84 , srv1960679316:84 , srv1993560338:84 , srv2092063459:84 , srv2107256971:84 , srv2111532206:84 , srv256134833:84 , srv262709016:84 , srv272702626:84 , srv313701950:84 , srv322782311:84 , srv332698058:84 , srv403627530:84 , srv412838327:84 , srv419520826:84 , srv421774562:84 , srv46557575:84 , srv480351088:84 , srv492607409:84 , srv503103567:84 , srv530674749:84 , srv530881598:84 , srv540866248:84 , srv607928680:84 , srv617605763:84 , srv617843363:84 , srv642859496:84 , srv647282321:84 , srv722559490:84 , srv737023316:84 , srv758369287:84 , srv759370070:84 , srv767644587:84 , srv791379749:84 , srv806681177:84 , srv837920766:84 , srv851017121:84 , srv853728802:84 , srv857245914:84 , srv889243276:84 , srv921521792:84 , srv960694513:84 , srv961033361:84 , srv1030761298:85 , srv1050859309:85 , srv1055875693:85 , srv119212329:85 , srv1281730714:85 , srv1529220363:85 , srv1608979671:85 , srv2040576814:85 , srv612723731:85 , srv86295221:683 } 2023-09-23 06:53:20,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:20,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906386455=52, srv2107256971=57, srv1960679316=53, srv1493150358=24, srv1229021248=16, srv851017121=92, srv480351088=70, srv1527288398=30, srv1448282240=22, srv1153883546=13, srv961033361=99, srv149360107=25, srv1516558149=29, srv185961857=48, srv1460952760=23, srv642859496=80, srv403627530=65, srv2111532206=58, srv612723731=77, srv791379749=89, srv1838793445=47, srv853728802=93, srv1281730714=19, srv732893009=84, srv1114486717=9, srv759370070=87, srv503103567=72, srv2092063459=56, srv1497514520=26, srv617605763=78, srv262709016=60, srv1032180724=3, srv156195638=32, srv607928680=76, srv647282321=81, srv11532620=12, srv1825578919=45, srv737023316=85, srv332698058=64, srv1147194960=11, srv119212329=15, srv1261427588=17, srv1621794055=38, srv272702626=61, srv767644587=88, srv1512798883=28, srv1565581232=33, srv889243276=96, srv1116497917=10, srv1030761298=2, srv1731202927=42, srv46557575=69, srv1618646630=36, srv1050859309=5, srv313701950=62, srv758369287=86, srv1016209812=1, srv1799012722=44, srv1837478273=46, srv86295221=95, srv1619074424=37, srv256134833=59, srv421774562=68, srv530881598=74, srv1675814496=40, srv412838327=66, srv857245914=94, srv1177096377=14, srv1370366172=20, srv682214246=82, srv1498590543=27, srv1529220363=31, srv1057183946=8, srv1273487486=18, srv419520826=67, srv1376444412=21, srv1869006553=49, srv492607409=71, srv530674749=73, srv1728799862=41, srv1586408404=34, srv837920766=91, srv322782311=63, srv960694513=98, srv1002113764=0, srv1055875693=7, srv617843363=79, srv1038802280=4, srv1608979671=35, srv722559490=83, srv1773616809=43, srv540866248=75, srv1993560338=54, srv921521792=97, srv2040576814=55, srv1632594360=39, srv1904442761=51, srv806681177=90, srv1051204705=6, srv1869857427=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:20,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:20,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:20,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 06:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 06:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 06:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 06:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 06:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 06:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 06:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 06:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 06:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 06:53:20,184 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:20,185 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1114486717:83 , srv1370366172:83 , srv1493150358:83 , srv1497514520:83 , srv1527288398:83 , srv1799012722:83 , srv682214246:83 , srv732893009:83 , srv1002113764:84 , srv1016209812:84 , srv1032180724:84 , srv1038802280:84 , srv1051204705:84 , srv1057183946:84 , srv1116497917:84 , srv1147194960:84 , srv11532620:84 , srv1153883546:84 , srv1177096377:84 , srv1229021248:84 , srv1261427588:84 , srv1273487486:84 , srv1376444412:84 , srv1448282240:84 , srv1460952760:84 , srv149360107:84 , srv1498590543:84 , srv1512798883:84 , srv1516558149:84 , srv156195638:84 , srv1565581232:84 , srv1586408404:84 , srv1618646630:84 , srv1619074424:84 , srv1621794055:84 , srv1632594360:84 , srv1675814496:84 , srv1728799862:84 , srv1731202927:84 , srv1773616809:84 , srv1825578919:84 , srv1837478273:84 , srv1838793445:84 , srv185961857:84 , srv1869006553:84 , srv1869857427:84 , srv1904442761:84 , srv1906386455:84 , srv1960679316:84 , srv1993560338:84 , srv2092063459:84 , srv2107256971:84 , srv2111532206:84 , srv256134833:84 , srv262709016:84 , srv272702626:84 , srv313701950:84 , srv322782311:84 , srv332698058:84 , srv403627530:84 , srv412838327:84 , srv419520826:84 , srv421774562:84 , srv46557575:84 , srv480351088:84 , srv492607409:84 , srv503103567:84 , srv530674749:84 , srv530881598:84 , srv540866248:84 , srv607928680:84 , srv617605763:84 , srv617843363:84 , srv642859496:84 , srv647282321:84 , srv722559490:84 , srv737023316:84 , srv758369287:84 , srv759370070:84 , srv767644587:84 , srv791379749:84 , srv806681177:84 , srv837920766:84 , srv851017121:84 , srv853728802:84 , srv857245914:84 , srv889243276:84 , srv921521792:84 , srv960694513:84 , srv961033361:84 , srv1030761298:85 , srv1050859309:85 , srv1055875693:85 , srv119212329:85 , srv1281730714:85 , srv1529220363:85 , srv1608979671:85 , srv2040576814:85 , srv612723731:85 , srv86295221:683 } 2023-09-23 06:53:20,211 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 322), ProcessCount=172 (was 173), AvailableMemoryMB=2602 (was 3099) 2023-09-23 06:53:20,222 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=172, AvailableMemoryMB=2602 2023-09-23 06:53:20,222 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 06:53:20,222 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-23 06:53:20,231 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1058341664:84 , srv1251658925:84 , srv1314121774:84 , srv1400522103:84 , srv158376249:84 , srv160483084:84 , srv1641010206:84 , srv1763953595:84 , srv1805905233:84 , srv1852318370:84 , srv2087808956:84 , srv2119362195:84 , srv23058092:84 , srv293693160:84 , srv379236146:84 , srv402477739:84 , srv421478623:84 , srv51570934:84 , srv519321492:84 , srv548391917:84 , srv556815248:84 , srv611936114:84 , srv62031843:84 , srv623325732:84 , srv633920888:84 , srv685038381:84 , srv782824412:84 , srv915935340:84 , srv919065129:84 , srv735511073:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-23 06:53:20,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:53:20,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1641010206=6, srv379236146=14, srv519321492=18, srv1314121774=2, srv1058341664=0, srv23058092=12, srv421478623=16, srv915935340=28, srv548391917=19, srv158376249=4, srv160483084=5, srv402477739=15, srv2087808956=10, srv2119362195=11, srv62031843=22, srv1251658925=1, srv782824412=27, srv919065129=29, srv633920888=24, srv51570934=17, srv623325732=23, srv556815248=20, srv735511073=26, srv293693160=13, srv611936114=21, srv1400522103=3, srv1805905233=8, srv1763953595=7, srv1852318370=9, srv685038381=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-23 06:53:20,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1686487522676203, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:53:20,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912724449975786, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1686487522676203, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-23 06:55:20,245 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6204944 different iterations. Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912724449975786 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:55:20,251 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1058341664:90 , srv1251658925:90 , srv1314121774:90 , srv1400522103:90 , srv158376249:90 , srv160483084:90 , srv1641010206:90 , srv1763953595:90 , srv1805905233:90 , srv1852318370:90 , srv2087808956:90 , srv2119362195:90 , srv23058092:90 , srv293693160:90 , srv379236146:90 , srv402477739:90 , srv421478623:90 , srv51570934:90 , srv519321492:90 , srv548391917:90 , srv556815248:90 , srv611936114:90 , srv62031843:90 , srv623325732:90 , srv633920888:90 , srv685038381:90 , srv735511073:90 , srv782824412:90 , srv915935340:90 , srv919065129:90 } 2023-09-23 06:55:20,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 06:55:20,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1641010206=6, srv379236146=14, srv519321492=18, srv1314121774=2, srv1058341664=0, srv23058092=12, srv421478623=16, srv915935340=28, srv548391917=19, srv158376249=4, srv160483084=5, srv402477739=15, srv2087808956=10, srv2119362195=11, srv62031843=22, srv1251658925=1, srv782824412=27, srv919065129=29, srv633920888=24, srv51570934=17, srv623325732=23, srv556815248=20, srv735511073=26, srv293693160=13, srv611936114=21, srv1400522103=3, srv1805905233=8, srv1763953595=7, srv1852318370=9, srv685038381=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 06:55:20,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 06:55:20,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-23 06:55:20,269 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 06:55:20,269 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1058341664:90 , srv1251658925:90 , srv1314121774:90 , srv1400522103:90 , srv158376249:90 , srv160483084:90 , srv1641010206:90 , srv1763953595:90 , srv1805905233:90 , srv1852318370:90 , srv2087808956:90 , srv2119362195:90 , srv23058092:90 , srv293693160:90 , srv379236146:90 , srv402477739:90 , srv421478623:90 , srv51570934:90 , srv519321492:90 , srv548391917:90 , srv556815248:90 , srv611936114:90 , srv62031843:90 , srv623325732:90 , srv633920888:90 , srv685038381:90 , srv735511073:90 , srv782824412:90 , srv915935340:90 , srv919065129:90 } 2023-09-23 06:55:20,281 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=141 (was 322), ProcessCount=171 (was 172), AvailableMemoryMB=1683 (was 2602)