2023-02-16 13:56:48,177 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-16 13:56:48,477 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-16 13:56:48,580 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-16 13:56:48,635 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-16 13:56:48,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-16 13:56:48,653 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=164, AvailableMemoryMB=4078 2023-02-16 13:56:48,656 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-16 13:56:48,657 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-16 13:56:48,769 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007815535:84 , srv1023056293:84 , srv1023970042:84 , srv1053873140:84 , srv1091738265:84 , srv1100739056:84 , srv1109211306:84 , srv1109406891:84 , srv1110132154:84 , srv1145409564:84 , srv1156582094:84 , srv1168891884:84 , srv1181342261:84 , srv119350818:84 , srv1220310591:84 , srv1224993599:84 , srv1246672308:84 , srv1290301486:84 , srv1310374838:84 , srv1318784308:84 , srv1399688212:84 , srv1401676746:84 , srv1405365022:84 , srv1418125233:84 , srv144161605:84 , srv1481499394:84 , srv1530682682:84 , srv1560314108:84 , srv1573208932:84 , srv1600647798:84 , srv1614840033:84 , srv1624512494:84 , srv1629184930:84 , srv1629508743:84 , srv1657334016:84 , srv1706691087:84 , srv1750110691:84 , srv1765367213:84 , srv1812443545:84 , srv1821014750:84 , srv1855602370:84 , srv1859919452:84 , srv1881874343:84 , srv1892096938:84 , srv1902086884:84 , srv1911344802:84 , srv1913368557:84 , srv19196748:84 , srv1935008290:84 , srv1952492988:84 , srv1956540803:84 , srv1971453889:84 , srv2018373244:84 , srv2027219705:84 , srv2040339045:84 , srv2055436152:84 , srv2059403752:84 , srv2068061222:84 , srv2079580564:84 , srv2120162963:84 , srv2121139860:84 , srv2126979817:84 , srv2138640190:84 , srv2140567417:84 , srv225250791:84 , srv298791434:84 , srv307362028:84 , srv308768287:84 , srv31746796:84 , srv354532694:84 , srv35718235:84 , srv361570570:84 , srv392515094:84 , srv405090556:84 , srv406904373:84 , srv413286373:84 , srv430842381:84 , srv487297698:84 , srv519298616:84 , srv522291322:84 , srv569468335:84 , srv621791443:84 , srv630290828:84 , srv631613172:84 , srv656248679:84 , srv666054011:84 , srv670476540:84 , srv678958608:84 , srv681790028:84 , srv763090752:84 , srv795577117:84 , srv805498056:84 , srv810317535:84 , srv829916400:84 , srv856242700:84 , srv868770367:84 , srv928788158:84 , srv929899409:84 , srv955639365:84 , srv942055277:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-16 13:56:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:48,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1892096938=43, srv1855602370=40, srv1145409564=9, srv1935008290=48, srv1109406891=7, srv1023970042=2, srv868770367=95, srv1530682682=26, srv1821014750=39, srv678958608=87, srv1399688212=20, srv1560314108=27, srv1053873140=3, srv1750110691=36, srv1110132154=8, srv1109211306=6, srv519298616=78, srv1956540803=50, srv307362028=66, srv119350818=13, srv522291322=79, srv1224993599=15, srv1911344802=45, srv1023056293=1, srv1401676746=21, srv763090752=89, srv631613172=83, srv405090556=73, srv1881874343=42, srv1971453889=51, srv430842381=76, srv2027219705=53, srv1629184930=32, srv1600647798=29, srv1859919452=41, srv955639365=99, srv487297698=77, srv1168891884=11, srv1624512494=31, srv2040339045=54, srv630290828=82, srv805498056=91, srv308768287=67, srv1156582094=10, srv31746796=68, srv392515094=72, srv1629508743=33, srv1573208932=28, srv19196748=47, srv2059403752=56, srv413286373=75, srv1812443545=38, srv810317535=92, srv1765367213=37, srv2138640190=62, srv1220310591=14, srv225250791=64, srv361570570=71, srv2126979817=61, srv1100739056=5, srv1657334016=34, srv354532694=69, srv856242700=94, srv35718235=70, srv1290301486=17, srv2079580564=58, srv1181342261=12, srv928788158=96, srv2120162963=59, srv829916400=93, srv2068061222=57, srv1091738265=4, srv942055277=98, srv1246672308=16, srv656248679=84, srv1706691087=35, srv929899409=97, srv1405365022=22, srv795577117=90, srv298791434=65, srv1902086884=44, srv681790028=88, srv2121139860=60, srv1007815535=0, srv2055436152=55, srv1418125233=23, srv1614840033=30, srv670476540=86, srv1481499394=25, srv2140567417=63, srv666054011=85, srv621791443=81, srv144161605=24, srv1952492988=49, srv2018373244=52, srv1318784308=19, srv406904373=74, srv1310374838=18, srv569468335=80, srv1913368557=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-16 13:56:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-16 13:56:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-16 13:56:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-16 13:56:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-16 13:56:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-16 13:56:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-16 13:56:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-16 13:56:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-16 13:56:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-16 13:56:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-16 13:56:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-16 13:56:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-16 13:56:49,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:49,019 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-16 13:56:49,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-16 13:56:49,439 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 531 ms to try 5000 different iterations. Found a solution that moves 4693 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17546075340770828. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040188792526880845, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059005982310038826, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5214444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040188792526880845, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.236, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:49,448 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv522291322:80 , srv1399688212:81 , srv1481499394:81 , srv1600647798:81 , srv19196748:81 , srv1952492988:81 , srv1023056293:82 , srv1100739056:82 , srv354532694:82 , srv1750110691:83 , srv225250791:83 , srv519298616:83 , srv955639365:83 , srv1023970042:84 , srv1109211306:84 , srv1401676746:84 , srv1657334016:84 , srv1765367213:84 , srv1812443545:84 , srv361570570:84 , srv631613172:84 , srv656248679:84 , srv666054011:84 , srv1007815535:85 , srv1109406891:85 , srv119350818:85 , srv144161605:85 , srv1624512494:85 , srv1855602370:85 , srv1935008290:85 , srv2055436152:85 , srv392515094:85 , srv487297698:85 , srv681790028:85 , srv763090752:85 , srv805498056:85 , srv1145409564:86 , srv1156582094:86 , srv1418125233:86 , srv1560314108:86 , srv1629184930:86 , srv1706691087:86 , srv1902086884:86 , srv2018373244:86 , srv2040339045:86 , srv2140567417:86 , srv307362028:86 , srv430842381:86 , srv621791443:86 , srv929899409:86 , srv1168891884:87 , srv1181342261:87 , srv1224993599:87 , srv1310374838:87 , srv1405365022:87 , srv1614840033:87 , srv1821014750:87 , srv1859919452:87 , srv1892096938:87 , srv1971453889:87 , srv2068061222:87 , srv2120162963:87 , srv2121139860:87 , srv2126979817:87 , srv31746796:87 , srv406904373:87 , srv569468335:87 , srv630290828:87 , srv795577117:87 , srv868770367:87 , srv1911344802:88 , srv1913368557:88 , srv1956540803:88 , srv2027219705:88 , srv2059403752:88 , srv308768287:88 , srv405090556:88 , srv670476540:88 , srv678958608:88 , srv1220310591:89 , srv1290301486:89 , srv1318784308:89 , srv1629508743:89 , srv2138640190:89 , srv810317535:89 , srv1530682682:90 , srv1573208932:90 , srv2079580564:90 , srv298791434:90 , srv35718235:90 , srv829916400:90 , srv856242700:90 , srv1091738265:91 , srv1246672308:91 , srv413286373:91 , srv1053873140:92 , srv1110132154:92 , srv928788158:93 , srv1881874343:95 , srv942055277:447 } 2023-02-16 13:56:49,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:49,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1892096938=43, srv1855602370=40, srv1145409564=9, srv1935008290=48, srv1109406891=7, srv1023970042=2, srv868770367=95, srv1530682682=26, srv1821014750=39, srv678958608=87, srv1399688212=20, srv1560314108=27, srv1053873140=3, srv1750110691=36, srv1110132154=8, srv1109211306=6, srv519298616=78, srv1956540803=50, srv307362028=66, srv119350818=13, srv522291322=79, srv1224993599=15, srv1911344802=45, srv1023056293=1, srv1401676746=21, srv763090752=89, srv631613172=83, srv405090556=73, srv1881874343=42, srv1971453889=51, srv430842381=76, srv2027219705=53, srv1629184930=32, srv1600647798=29, srv1859919452=41, srv955639365=99, srv487297698=77, srv1168891884=11, srv1624512494=31, srv2040339045=54, srv630290828=82, srv805498056=91, srv308768287=67, srv1156582094=10, srv31746796=68, srv392515094=72, srv1629508743=33, srv1573208932=28, srv19196748=47, srv2059403752=56, srv413286373=75, srv1812443545=38, srv810317535=92, srv1765367213=37, srv2138640190=62, srv1220310591=14, srv225250791=64, srv361570570=71, srv2126979817=61, srv1100739056=5, srv1657334016=34, srv354532694=69, srv856242700=94, srv35718235=70, srv1290301486=17, srv2079580564=58, srv1181342261=12, srv928788158=96, srv2120162963=59, srv829916400=93, srv2068061222=57, srv1091738265=4, srv942055277=98, srv1246672308=16, srv656248679=84, srv1706691087=35, srv929899409=97, srv1405365022=22, srv795577117=90, srv298791434=65, srv1902086884=44, srv681790028=88, srv2121139860=60, srv1007815535=0, srv2055436152=55, srv1418125233=23, srv1614840033=30, srv670476540=86, srv1481499394=25, srv2140567417=63, srv666054011=85, srv621791443=81, srv144161605=24, srv1952492988=49, srv2018373244=52, srv1318784308=19, srv406904373=74, srv1310374838=18, srv569468335=80, srv1913368557=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:49,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:49,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:49,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:49,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:49,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:49,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:49,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-16 13:56:49,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-16 13:56:49,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-16 13:56:49,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-16 13:56:49,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-16 13:56:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-16 13:56:49,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-16 13:56:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-16 13:56:49,586 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040188792526880845, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059005982310038826, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040188792526880845, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.236, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:49,586 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-16 13:56:49,586 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17542789231935416, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040188792526880845, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059005982310038826, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040188792526880845, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.236, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-16 13:56:49,796 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 295 ms to try 5000 different iterations. Found a solution that moves 2628 regions; Going from a computed imbalance of 0.17542789231935416 to a new imbalance of 0.014491922206722897. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022828978069128678); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037590146045017515, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.292, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022828978069128678); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:49,804 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv928788158:78 , srv1881874343:79 , srv1110132154:80 , srv829916400:80 , srv1091738265:81 , srv1220310591:82 , srv1573208932:82 , srv2059403752:82 , srv2079580564:82 , srv298791434:82 , srv308768287:82 , srv35718235:82 , srv413286373:82 , srv1290301486:83 , srv1318784308:83 , srv1530682682:83 , srv1956540803:83 , srv2121139860:83 , srv405090556:83 , srv1053873140:84 , srv1168891884:84 , srv1246672308:84 , srv1310374838:84 , srv1560314108:84 , srv1629184930:84 , srv1629508743:84 , srv1902086884:84 , srv1913368557:84 , srv2027219705:84 , srv2120162963:84 , srv2126979817:84 , srv2138640190:84 , srv31746796:84 , srv630290828:84 , srv670476540:84 , srv856242700:84 , srv1109406891:85 , srv1145409564:85 , srv1181342261:85 , srv1418125233:85 , srv1614840033:85 , srv1706691087:85 , srv1821014750:85 , srv1892096938:85 , srv1911344802:85 , srv2018373244:85 , srv2068061222:85 , srv307362028:85 , srv569468335:85 , srv631613172:85 , srv678958608:85 , srv795577117:85 , srv929899409:85 , srv119350818:86 , srv1224993599:86 , srv1405365022:86 , srv1765367213:86 , srv1859919452:86 , srv1935008290:86 , srv2040339045:86 , srv2140567417:86 , srv392515094:86 , srv621791443:86 , srv810317535:86 , srv868770367:86 , srv1007815535:87 , srv1156582094:87 , srv1855602370:87 , srv1971453889:87 , srv430842381:87 , srv519298616:87 , srv666054011:87 , srv681790028:87 , srv1023970042:88 , srv1100739056:88 , srv1401676746:88 , srv144161605:88 , srv1750110691:88 , srv2055436152:88 , srv406904373:88 , srv487297698:88 , srv656248679:88 , srv763090752:88 , srv805498056:88 , srv1109211306:89 , srv1399688212:89 , srv1624512494:89 , srv1657334016:89 , srv1952492988:89 , srv361570570:89 , srv955639365:89 , srv1812443545:90 , srv225250791:90 , srv354532694:90 , srv1023056293:91 , srv1481499394:91 , srv1600647798:91 , srv19196748:91 , srv522291322:92 , srv942055277:530 } 2023-02-16 13:56:49,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:49,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1892096938=43, srv1855602370=40, srv1145409564=9, srv1935008290=48, srv1109406891=7, srv1023970042=2, srv868770367=95, srv1530682682=26, srv1821014750=39, srv678958608=87, srv1399688212=20, srv1560314108=27, srv1053873140=3, srv1750110691=36, srv1110132154=8, srv1109211306=6, srv519298616=78, srv1956540803=50, srv307362028=66, srv119350818=13, srv522291322=79, srv1224993599=15, srv1911344802=45, srv1023056293=1, srv1401676746=21, srv763090752=89, srv631613172=83, srv405090556=73, srv1881874343=42, srv1971453889=51, srv430842381=76, srv2027219705=53, srv1629184930=32, srv1600647798=29, srv1859919452=41, srv955639365=99, srv487297698=77, srv1168891884=11, srv1624512494=31, srv2040339045=54, srv630290828=82, srv805498056=91, srv308768287=67, srv1156582094=10, srv31746796=68, srv392515094=72, srv1629508743=33, srv1573208932=28, srv19196748=47, srv2059403752=56, srv413286373=75, srv1812443545=38, srv810317535=92, srv1765367213=37, srv2138640190=62, srv1220310591=14, srv225250791=64, srv361570570=71, srv2126979817=61, srv1100739056=5, srv1657334016=34, srv354532694=69, srv856242700=94, srv35718235=70, srv1290301486=17, srv2079580564=58, srv1181342261=12, srv928788158=96, srv2120162963=59, srv829916400=93, srv2068061222=57, srv1091738265=4, srv942055277=98, srv1246672308=16, srv656248679=84, srv1706691087=35, srv929899409=97, srv1405365022=22, srv795577117=90, srv298791434=65, srv1902086884=44, srv681790028=88, srv2121139860=60, srv1007815535=0, srv2055436152=55, srv1418125233=23, srv1614840033=30, srv670476540=86, srv1481499394=25, srv2140567417=63, srv666054011=85, srv621791443=81, srv144161605=24, srv1952492988=49, srv2018373244=52, srv1318784308=19, srv406904373=74, srv1310374838=18, srv569468335=80, srv1913368557=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:49,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:49,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:49,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:49,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:49,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:49,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:49,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:49,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:49,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-16 13:56:49,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-16 13:56:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-16 13:56:49,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-16 13:56:49,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-16 13:56:49,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-16 13:56:49,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-16 13:56:49,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-16 13:56:49,929 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022828978069128678); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037590146045017515, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022828978069128678); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:49,930 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-16 13:56:49,930 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014473520557416561, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022828978069128678); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037590146045017515, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022828978069128678); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-16 13:56:50,093 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 241 ms to try 5000 different iterations. Found a solution that moves 976 regions; Going from a computed imbalance of 0.014473520557416561 to a new imbalance of 8.053264488400799E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063269209666960285); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008951797592365465); MoveCostFunction : (multiplier=7.0, imbalance=0.10844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063269209666960285); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,094 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1053873140:82 , srv406904373:82 , srv1246672308:83 , srv1812443545:83 , srv1881874343:83 , srv810317535:83 , srv1023056293:84 , srv1109211306:84 , srv1156582094:84 , srv1224993599:84 , srv1405365022:84 , srv144161605:84 , srv1530682682:84 , srv1614840033:84 , srv1624512494:84 , srv1657334016:84 , srv1913368557:84 , srv1971453889:84 , srv2055436152:84 , srv2138640190:84 , srv361570570:84 , srv413286373:84 , srv522291322:84 , srv678958608:84 , srv1023970042:85 , srv1145409564:85 , srv1318784308:85 , srv1401676746:85 , srv1481499394:85 , srv1573208932:85 , srv1600647798:85 , srv1629508743:85 , srv1821014750:85 , srv1855602370:85 , srv1859919452:85 , srv1911344802:85 , srv2027219705:85 , srv2040339045:85 , srv225250791:85 , srv354532694:85 , srv430842381:85 , srv569468335:85 , srv621791443:85 , srv763090752:85 , srv795577117:85 , srv805498056:85 , srv856242700:85 , srv868770367:85 , srv929899409:85 , srv1007815535:86 , srv1091738265:86 , srv1110132154:86 , srv1168891884:86 , srv1181342261:86 , srv1220310591:86 , srv1750110691:86 , srv1892096938:86 , srv1956540803:86 , srv2018373244:86 , srv2068061222:86 , srv2079580564:86 , srv2120162963:86 , srv2126979817:86 , srv2140567417:86 , srv298791434:86 , srv307362028:86 , srv35718235:86 , srv392515094:86 , srv487297698:86 , srv656248679:86 , srv666054011:86 , srv670476540:86 , srv681790028:86 , srv928788158:86 , srv955639365:86 , srv1100739056:87 , srv119350818:87 , srv1290301486:87 , srv1310374838:87 , srv1399688212:87 , srv1418125233:87 , srv1560314108:87 , srv1629184930:87 , srv1706691087:87 , srv1902086884:87 , srv19196748:87 , srv1935008290:87 , srv1952492988:87 , srv2121139860:87 , srv31746796:87 , srv405090556:87 , srv630290828:87 , srv829916400:87 , srv1109406891:88 , srv1765367213:88 , srv2059403752:88 , srv308768287:88 , srv519298616:88 , srv631613172:88 , srv942055277:537 } 2023-02-16 13:56:50,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:50,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1892096938=43, srv1855602370=40, srv1145409564=9, srv1935008290=48, srv1109406891=7, srv1023970042=2, srv868770367=95, srv1530682682=26, srv1821014750=39, srv678958608=87, srv1399688212=20, srv1560314108=27, srv1053873140=3, srv1750110691=36, srv1110132154=8, srv1109211306=6, srv519298616=78, srv1956540803=50, srv307362028=66, srv119350818=13, srv522291322=79, srv1224993599=15, srv1911344802=45, srv1023056293=1, srv1401676746=21, srv763090752=89, srv631613172=83, srv405090556=73, srv1881874343=42, srv1971453889=51, srv430842381=76, srv2027219705=53, srv1629184930=32, srv1600647798=29, srv1859919452=41, srv955639365=99, srv487297698=77, srv1168891884=11, srv1624512494=31, srv2040339045=54, srv630290828=82, srv805498056=91, srv308768287=67, srv1156582094=10, srv31746796=68, srv392515094=72, srv1629508743=33, srv1573208932=28, srv19196748=47, srv2059403752=56, srv413286373=75, srv1812443545=38, srv810317535=92, srv1765367213=37, srv2138640190=62, srv1220310591=14, srv225250791=64, srv361570570=71, srv2126979817=61, srv1100739056=5, srv1657334016=34, srv354532694=69, srv856242700=94, srv35718235=70, srv1290301486=17, srv2079580564=58, srv1181342261=12, srv928788158=96, srv2120162963=59, srv829916400=93, srv2068061222=57, srv1091738265=4, srv942055277=98, srv1246672308=16, srv656248679=84, srv1706691087=35, srv929899409=97, srv1405365022=22, srv795577117=90, srv298791434=65, srv1902086884=44, srv681790028=88, srv2121139860=60, srv1007815535=0, srv2055436152=55, srv1418125233=23, srv1614840033=30, srv670476540=86, srv1481499394=25, srv2140567417=63, srv666054011=85, srv621791443=81, srv144161605=24, srv1952492988=49, srv2018373244=52, srv1318784308=19, srv406904373=74, srv1310374838=18, srv569468335=80, srv1913368557=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:50,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:50,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-16 13:56:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-16 13:56:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-16 13:56:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-16 13:56:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-16 13:56:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-16 13:56:50,191 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063269209666960285); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008951797592365465); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063269209666960285); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,191 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-16 13:56:50,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.98492350771973E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063269209666960285); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008951797592365465); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063269209666960285); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-16 13:56:50,408 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 289 ms to try 5000 different iterations. Found a solution that moves 276 regions; Going from a computed imbalance of 7.98492350771973E-4 to a new imbalance of 3.601040971934877E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.030666666666666665, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,410 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1091738265:83 , srv1290301486:83 , srv19196748:83 , srv2068061222:83 , srv856242700:83 , srv1007815535:84 , srv1053873140:84 , srv1156582094:84 , srv1181342261:84 , srv119350818:84 , srv1220310591:84 , srv1246672308:84 , srv1310374838:84 , srv1418125233:84 , srv1624512494:84 , srv1629508743:84 , srv1706691087:84 , srv1859919452:84 , srv1892096938:84 , srv1902086884:84 , srv1911344802:84 , srv1935008290:84 , srv1956540803:84 , srv1971453889:84 , srv2059403752:84 , srv2079580564:84 , srv2121139860:84 , srv2140567417:84 , srv225250791:84 , srv307362028:84 , srv308768287:84 , srv31746796:84 , srv35718235:84 , srv361570570:84 , srv430842381:84 , srv487297698:84 , srv519298616:84 , srv630290828:84 , srv656248679:84 , srv670476540:84 , srv681790028:84 , srv763090752:84 , srv805498056:84 , srv868770367:84 , srv955639365:84 , srv1023056293:85 , srv1023970042:85 , srv1100739056:85 , srv1109211306:85 , srv1109406891:85 , srv1110132154:85 , srv1168891884:85 , srv1224993599:85 , srv1318784308:85 , srv1399688212:85 , srv1401676746:85 , srv1405365022:85 , srv144161605:85 , srv1481499394:85 , srv1530682682:85 , srv1560314108:85 , srv1600647798:85 , srv1614840033:85 , srv1629184930:85 , srv1657334016:85 , srv1750110691:85 , srv1765367213:85 , srv1812443545:85 , srv1821014750:85 , srv1855602370:85 , srv1881874343:85 , srv1913368557:85 , srv1952492988:85 , srv2018373244:85 , srv2027219705:85 , srv2040339045:85 , srv2055436152:85 , srv2120162963:85 , srv2126979817:85 , srv2138640190:85 , srv298791434:85 , srv392515094:85 , srv405090556:85 , srv406904373:85 , srv413286373:85 , srv569468335:85 , srv621791443:85 , srv631613172:85 , srv666054011:85 , srv678958608:85 , srv795577117:85 , srv810317535:85 , srv829916400:85 , srv928788158:85 , srv1145409564:86 , srv1573208932:86 , srv354532694:86 , srv929899409:86 , srv522291322:87 , srv942055277:629 } 2023-02-16 13:56:50,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:50,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1892096938=43, srv1855602370=40, srv1145409564=9, srv1935008290=48, srv1109406891=7, srv1023970042=2, srv868770367=95, srv1530682682=26, srv1821014750=39, srv678958608=87, srv1399688212=20, srv1560314108=27, srv1053873140=3, srv1750110691=36, srv1110132154=8, srv1109211306=6, srv519298616=78, srv1956540803=50, srv307362028=66, srv119350818=13, srv522291322=79, srv1224993599=15, srv1911344802=45, srv1023056293=1, srv1401676746=21, srv763090752=89, srv631613172=83, srv405090556=73, srv1881874343=42, srv1971453889=51, srv430842381=76, srv2027219705=53, srv1629184930=32, srv1600647798=29, srv1859919452=41, srv955639365=99, srv487297698=77, srv1168891884=11, srv1624512494=31, srv2040339045=54, srv630290828=82, srv805498056=91, srv308768287=67, srv1156582094=10, srv31746796=68, srv392515094=72, srv1629508743=33, srv1573208932=28, srv19196748=47, srv2059403752=56, srv413286373=75, srv1812443545=38, srv810317535=92, srv1765367213=37, srv2138640190=62, srv1220310591=14, srv225250791=64, srv361570570=71, srv2126979817=61, srv1100739056=5, srv1657334016=34, srv354532694=69, srv856242700=94, srv35718235=70, srv1290301486=17, srv2079580564=58, srv1181342261=12, srv928788158=96, srv2120162963=59, srv829916400=93, srv2068061222=57, srv1091738265=4, srv942055277=98, srv1246672308=16, srv656248679=84, srv1706691087=35, srv929899409=97, srv1405365022=22, srv795577117=90, srv298791434=65, srv1902086884=44, srv681790028=88, srv2121139860=60, srv1007815535=0, srv2055436152=55, srv1418125233=23, srv1614840033=30, srv670476540=86, srv1481499394=25, srv2140567417=63, srv666054011=85, srv621791443=81, srv144161605=24, srv1952492988=49, srv2018373244=52, srv1318784308=19, srv406904373=74, srv1310374838=18, srv569468335=80, srv1913368557=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-16 13:56:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-16 13:56:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-16 13:56:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-16 13:56:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-16 13:56:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-16 13:56:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-16 13:56:50,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,508 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-16 13:56:50,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.407781641320378E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-16 13:56:50,650 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 214 ms to try 5000 different iterations. Found a solution that moves 34 regions; Going from a computed imbalance of 3.407781641320378E-5 to a new imbalance of 6.6360587607078495E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.003777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,650 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1109406891:83 , srv1110132154:83 , srv1573208932:83 , srv1765367213:83 , srv298791434:83 , srv354532694:83 , srv405090556:83 , srv487297698:83 , srv522291322:83 , srv810317535:83 , srv1007815535:84 , srv1023056293:84 , srv1023970042:84 , srv1053873140:84 , srv1100739056:84 , srv1109211306:84 , srv1145409564:84 , srv1168891884:84 , srv1181342261:84 , srv119350818:84 , srv1224993599:84 , srv1246672308:84 , srv1290301486:84 , srv1310374838:84 , srv1318784308:84 , srv1399688212:84 , srv1401676746:84 , srv1405365022:84 , srv1418125233:84 , srv144161605:84 , srv1481499394:84 , srv1530682682:84 , srv1560314108:84 , srv1600647798:84 , srv1624512494:84 , srv1629184930:84 , srv1629508743:84 , srv1657334016:84 , srv1706691087:84 , srv1750110691:84 , srv1812443545:84 , srv1821014750:84 , srv1855602370:84 , srv1859919452:84 , srv1881874343:84 , srv1892096938:84 , srv1911344802:84 , srv19196748:84 , srv1935008290:84 , srv1952492988:84 , srv1971453889:84 , srv2018373244:84 , srv2027219705:84 , srv2040339045:84 , srv2055436152:84 , srv2059403752:84 , srv2079580564:84 , srv2120162963:84 , srv2126979817:84 , srv2138640190:84 , srv2140567417:84 , srv225250791:84 , srv308768287:84 , srv31746796:84 , srv35718235:84 , srv392515094:84 , srv406904373:84 , srv413286373:84 , srv430842381:84 , srv519298616:84 , srv569468335:84 , srv621791443:84 , srv630290828:84 , srv631613172:84 , srv656248679:84 , srv666054011:84 , srv670476540:84 , srv678958608:84 , srv681790028:84 , srv763090752:84 , srv795577117:84 , srv805498056:84 , srv829916400:84 , srv856242700:84 , srv868770367:84 , srv928788158:84 , srv929899409:84 , srv955639365:84 , srv1091738265:85 , srv1156582094:85 , srv1220310591:85 , srv1614840033:85 , srv1902086884:85 , srv1913368557:85 , srv1956540803:85 , srv2068061222:85 , srv2121139860:85 , srv307362028:85 , srv361570570:85 , srv942055277:683 } 2023-02-16 13:56:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:50,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1892096938=43, srv1855602370=40, srv1145409564=9, srv1935008290=48, srv1109406891=7, srv1023970042=2, srv868770367=95, srv1530682682=26, srv1821014750=39, srv678958608=87, srv1399688212=20, srv1560314108=27, srv1053873140=3, srv1750110691=36, srv1110132154=8, srv1109211306=6, srv519298616=78, srv1956540803=50, srv307362028=66, srv119350818=13, srv522291322=79, srv1224993599=15, srv1911344802=45, srv1023056293=1, srv1401676746=21, srv763090752=89, srv631613172=83, srv405090556=73, srv1881874343=42, srv1971453889=51, srv430842381=76, srv2027219705=53, srv1629184930=32, srv1600647798=29, srv1859919452=41, srv955639365=99, srv487297698=77, srv1168891884=11, srv1624512494=31, srv2040339045=54, srv630290828=82, srv805498056=91, srv308768287=67, srv1156582094=10, srv31746796=68, srv392515094=72, srv1629508743=33, srv1573208932=28, srv19196748=47, srv2059403752=56, srv413286373=75, srv1812443545=38, srv810317535=92, srv1765367213=37, srv2138640190=62, srv1220310591=14, srv225250791=64, srv361570570=71, srv2126979817=61, srv1100739056=5, srv1657334016=34, srv354532694=69, srv856242700=94, srv35718235=70, srv1290301486=17, srv2079580564=58, srv1181342261=12, srv928788158=96, srv2120162963=59, srv829916400=93, srv2068061222=57, srv1091738265=4, srv942055277=98, srv1246672308=16, srv656248679=84, srv1706691087=35, srv929899409=97, srv1405365022=22, srv795577117=90, srv298791434=65, srv1902086884=44, srv681790028=88, srv2121139860=60, srv1007815535=0, srv2055436152=55, srv1418125233=23, srv1614840033=30, srv670476540=86, srv1481499394=25, srv2140567417=63, srv666054011=85, srv621791443=81, srv144161605=24, srv1952492988=49, srv2018373244=52, srv1318784308=19, srv406904373=74, srv1310374838=18, srv569468335=80, srv1913368557=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:50,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-16 13:56:50,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-16 13:56:50,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-16 13:56:50,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-16 13:56:50,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-16 13:56:50,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-16 13:56:50,753 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,754 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1109406891:83 , srv1110132154:83 , srv1573208932:83 , srv1765367213:83 , srv298791434:83 , srv354532694:83 , srv405090556:83 , srv487297698:83 , srv522291322:83 , srv810317535:83 , srv1007815535:84 , srv1023056293:84 , srv1023970042:84 , srv1053873140:84 , srv1100739056:84 , srv1109211306:84 , srv1145409564:84 , srv1168891884:84 , srv1181342261:84 , srv119350818:84 , srv1224993599:84 , srv1246672308:84 , srv1290301486:84 , srv1310374838:84 , srv1318784308:84 , srv1399688212:84 , srv1401676746:84 , srv1405365022:84 , srv1418125233:84 , srv144161605:84 , srv1481499394:84 , srv1530682682:84 , srv1560314108:84 , srv1600647798:84 , srv1624512494:84 , srv1629184930:84 , srv1629508743:84 , srv1657334016:84 , srv1706691087:84 , srv1750110691:84 , srv1812443545:84 , srv1821014750:84 , srv1855602370:84 , srv1859919452:84 , srv1881874343:84 , srv1892096938:84 , srv1911344802:84 , srv19196748:84 , srv1935008290:84 , srv1952492988:84 , srv1971453889:84 , srv2018373244:84 , srv2027219705:84 , srv2040339045:84 , srv2055436152:84 , srv2059403752:84 , srv2079580564:84 , srv2120162963:84 , srv2126979817:84 , srv2138640190:84 , srv2140567417:84 , srv225250791:84 , srv308768287:84 , srv31746796:84 , srv35718235:84 , srv392515094:84 , srv406904373:84 , srv413286373:84 , srv430842381:84 , srv519298616:84 , srv569468335:84 , srv621791443:84 , srv630290828:84 , srv631613172:84 , srv656248679:84 , srv666054011:84 , srv670476540:84 , srv678958608:84 , srv681790028:84 , srv763090752:84 , srv795577117:84 , srv805498056:84 , srv829916400:84 , srv856242700:84 , srv868770367:84 , srv928788158:84 , srv929899409:84 , srv955639365:84 , srv1091738265:85 , srv1156582094:85 , srv1220310591:85 , srv1614840033:85 , srv1902086884:85 , srv1913368557:85 , srv1956540803:85 , srv2068061222:85 , srv2121139860:85 , srv307362028:85 , srv361570570:85 , srv942055277:683 } 2023-02-16 13:56:50,779 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 357) - SystemLoadAverage LEAK? -, ProcessCount=164 (was 164), AvailableMemoryMB=3594 (was 4078) 2023-02-16 13:56:50,786 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=164, AvailableMemoryMB=3594 2023-02-16 13:56:50,787 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-16 13:56:50,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-16 13:56:50,795 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1118751664:84 , srv1132705083:84 , srv1189644610:84 , srv1469745189:84 , srv1655129848:84 , srv1658795783:84 , srv1863996617:84 , srv1908472651:84 , srv1910020335:84 , srv1949213273:84 , srv1974554667:84 , srv1978330696:84 , srv2052335305:84 , srv2065607599:84 , srv2092503383:84 , srv2102868051:84 , srv263914350:84 , srv523395245:84 , srv55034708:84 , srv577631847:84 , srv609460322:84 , srv611421637:84 , srv652761394:84 , srv772842644:84 , srv832014215:84 , srv870281467:84 , srv952632021:84 , srv965987882:84 , srv99407283:84 , srv1355679624:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-16 13:56:50,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:56:50,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263914350=17, srv965987882=28, srv2052335305=13, srv1658795783=6, srv1132705083=1, srv2092503383=15, srv1910020335=9, srv1949213273=10, srv1978330696=12, srv1908472651=8, srv609460322=21, srv1974554667=11, srv832014215=25, srv1189644610=2, srv1655129848=5, srv1118751664=0, srv952632021=27, srv870281467=26, srv523395245=18, srv1863996617=7, srv577631847=20, srv611421637=22, srv2102868051=16, srv2065607599=14, srv55034708=19, srv772842644=24, srv99407283=29, srv652761394=23, srv1355679624=3, srv1469745189=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:56:50,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:56:50,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:56:50,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:56:50,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-16 13:56:50,818 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1364777274115896, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:56:50,819 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912623080126945, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1364777274115896, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-16 13:58:50,808 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 10104509 different iterations. Found a solution that moves 1842 regions; Going from a computed imbalance of 0.9912623080126945 to a new imbalance of 4.2993198912066E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:58:50,816 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1118751664:90 , srv1132705083:90 , srv1189644610:90 , srv1355679624:90 , srv1469745189:90 , srv1655129848:90 , srv1658795783:90 , srv1863996617:90 , srv1908472651:90 , srv1910020335:90 , srv1949213273:90 , srv1974554667:90 , srv1978330696:90 , srv2052335305:90 , srv2065607599:90 , srv2092503383:90 , srv2102868051:90 , srv263914350:90 , srv523395245:90 , srv55034708:90 , srv577631847:90 , srv609460322:90 , srv611421637:90 , srv652761394:90 , srv772842644:90 , srv832014215:90 , srv870281467:90 , srv952632021:90 , srv965987882:90 , srv99407283:90 } 2023-02-16 13:58:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-16 13:58:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263914350=17, srv965987882=28, srv2052335305=13, srv1658795783=6, srv1132705083=1, srv2092503383=15, srv1910020335=9, srv1949213273=10, srv1978330696=12, srv1908472651=8, srv609460322=21, srv1974554667=11, srv832014215=25, srv1189644610=2, srv1655129848=5, srv1118751664=0, srv952632021=27, srv870281467=26, srv523395245=18, srv1863996617=7, srv577631847=20, srv611421637=22, srv2102868051=16, srv2065607599=14, srv55034708=19, srv772842644=24, srv99407283=29, srv652761394=23, srv1355679624=3, srv1469745189=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-16 13:58:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-16 13:58:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-16 13:58:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-16 13:58:50,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-16 13:58:50,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-16 13:58:50,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-16 13:58:50,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-16 13:58:50,838 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-16 13:58:50,839 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1118751664:90 , srv1132705083:90 , srv1189644610:90 , srv1355679624:90 , srv1469745189:90 , srv1655129848:90 , srv1658795783:90 , srv1863996617:90 , srv1908472651:90 , srv1910020335:90 , srv1949213273:90 , srv1974554667:90 , srv1978330696:90 , srv2052335305:90 , srv2065607599:90 , srv2092503383:90 , srv2102868051:90 , srv263914350:90 , srv523395245:90 , srv55034708:90 , srv577631847:90 , srv609460322:90 , srv611421637:90 , srv652761394:90 , srv772842644:90 , srv832014215:90 , srv870281467:90 , srv952632021:90 , srv965987882:90 , srv99407283:90 } 2023-02-16 13:58:50,855 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=245 (was 360), ProcessCount=164 (was 164), AvailableMemoryMB=3851 (was 3594) - AvailableMemoryMB LEAK? -