2023-08-05 06:53:11,219 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-05 06:53:11,546 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-05 06:53:11,663 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-05 06:53:11,727 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-05 06:53:11,735 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-05 06:53:11,749 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=305, ProcessCount=173, AvailableMemoryMB=2067 2023-08-05 06:53:11,751 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-05 06:53:11,751 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-05 06:53:11,890 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1036315820:84 , srv1046572669:84 , srv1057095530:84 , srv1057672839:84 , srv1063934431:84 , srv1095460410:84 , srv1118667847:84 , srv1128754426:84 , srv1139596000:84 , srv1144448281:84 , srv1161952646:84 , srv1173710940:84 , srv122521629:84 , srv123503810:84 , srv125206130:84 , srv1261439483:84 , srv132975609:84 , srv1341974060:84 , srv1359265432:84 , srv1364803629:84 , srv1380763891:84 , srv1403099424:84 , srv1425186077:84 , srv1471147673:84 , srv1537325140:84 , srv1560274941:84 , srv1575680459:84 , srv1592106342:84 , srv1633168401:84 , srv166912145:84 , srv1728060904:84 , srv1793093060:84 , srv1819199192:84 , srv1864015269:84 , srv1870833286:84 , srv188373002:84 , srv189062829:84 , srv1892953847:84 , srv1920405275:84 , srv1926511940:84 , srv193170916:84 , srv1961196575:84 , srv1967118206:84 , srv1976753015:84 , srv1981561157:84 , srv2007567041:84 , srv2045698210:84 , srv2046124697:84 , srv2074832508:84 , srv2077445258:84 , srv2091361836:84 , srv2106026262:84 , srv2147363000:84 , srv248521588:84 , srv255544890:84 , srv255962218:84 , srv271126469:84 , srv281233062:84 , srv28548298:84 , srv286075388:84 , srv314858622:84 , srv335020279:84 , srv341857780:84 , srv388873011:84 , srv399483279:84 , srv442883428:84 , srv480530053:84 , srv500348082:84 , srv50395223:84 , srv525487263:84 , srv530437413:84 , srv593544474:84 , srv597206712:84 , srv601286307:84 , srv631925823:84 , srv660658865:84 , srv670273939:84 , srv67520564:84 , srv72333393:84 , srv733015498:84 , srv746862247:84 , srv769982771:84 , srv800107385:84 , srv838726459:84 , srv884492301:84 , srv894739427:84 , srv897372335:84 , srv902000959:84 , srv916236091:84 , srv919737266:84 , srv923546395:84 , srv942938409:84 , srv947386842:84 , srv952860107:84 , srv96905327:84 , srv969959278:84 , srv97394816:84 , srv977639761:84 , srv978900089:84 , srv1862632179:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-05 06:53:11,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1128754426=7, srv335020279=62, srv1967118206=43, srv132975609=16, srv1471147673=23, srv1633168401=28, srv1118667847=6, srv1793093060=31, srv1046572669=1, srv593544474=72, srv1359265432=18, srv2106026262=52, srv1261439483=15, srv530437413=71, srv72333393=79, srv1341974060=17, srv947386842=93, srv399483279=65, srv952860107=94, srv916236091=89, srv977639761=98, srv1173710940=11, srv631925823=75, srv166912145=29, srv525487263=70, srv769982771=82, srv2045698210=47, srv1139596000=8, srv1862632179=33, srv442883428=66, srv884492301=85, srv894739427=86, srv1864015269=34, srv50395223=69, srv1592106342=27, srv733015498=80, srv123503810=13, srv1976753015=44, srv838726459=84, srv2046124697=48, srv122521629=12, srv1144448281=9, srv1537325140=24, srv189062829=37, srv480530053=67, srv1057672839=3, srv1036315820=0, srv97394816=97, srv1819199192=32, srv1057095530=2, srv1728060904=30, srv1063934431=4, srv1380763891=20, srv281233062=58, srv286075388=60, srv1926511940=40, srv601286307=74, srv660658865=76, srv1920405275=39, srv193170916=41, srv902000959=88, srv923546395=91, srv670273939=77, srv67520564=78, srv942938409=92, srv978900089=99, srv1560274941=25, srv2147363000=53, srv314858622=61, srv341857780=63, srv800107385=83, srv897372335=87, srv1892953847=38, srv2074832508=49, srv2007567041=46, srv969959278=96, srv388873011=64, srv188373002=36, srv2077445258=50, srv746862247=81, srv255962218=56, srv1961196575=42, srv1095460410=5, srv28548298=59, srv597206712=73, srv1403099424=21, srv271126469=57, srv1575680459=26, srv1161952646=10, srv1425186077=22, srv919737266=90, srv1364803629=19, srv248521588=54, srv2091361836=51, srv1870833286=35, srv1981561157=45, srv500348082=68, srv125206130=14, srv255544890=55, srv96905327=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 06:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 06:53:11,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 06:53:11,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 06:53:11,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 06:53:11,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 06:53:11,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 06:53:11,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 06:53:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 06:53:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 06:53:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 06:53:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 06:53:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 06:53:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 06:53:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 06:53:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 06:53:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 06:53:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 06:53:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 06:53:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 06:53:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 06:53:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 06:53:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 06:53:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 06:53:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 06:53:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 06:53:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 06:53:12,215 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:12,215 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 06:53:12,216 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 06:53:12,687 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 658 ms to try 5000 different iterations. Found a solution that moves 4709 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17463599906634342. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0395135024064937, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05648117789839482, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5232222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0395135024064937, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16975, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.237, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:12,713 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv480530053:80 , srv188373002:81 , srv2077445258:81 , srv800107385:81 , srv67520564:82 , srv838726459:82 , srv894739427:82 , srv1118667847:83 , srv1920405275:83 , srv1967118206:83 , srv525487263:83 , srv660658865:83 , srv978900089:83 , srv1036315820:84 , srv1728060904:84 , srv1892953847:84 , srv1926511940:84 , srv2106026262:84 , srv593544474:84 , srv631925823:84 , srv902000959:84 , srv969959278:84 , srv1057672839:85 , srv1095460410:85 , srv125206130:85 , srv1359265432:85 , srv1633168401:85 , srv1864015269:85 , srv2007567041:85 , srv2046124697:85 , srv255544890:85 , srv255962218:85 , srv314858622:85 , srv442883428:85 , srv500348082:85 , srv769982771:85 , srv919737266:85 , srv977639761:85 , srv1144448281:86 , srv1364803629:86 , srv1380763891:86 , srv1870833286:86 , srv28548298:86 , srv388873011:86 , srv399483279:86 , srv50395223:86 , srv601286307:86 , srv916236091:86 , srv923546395:86 , srv942938409:86 , srv96905327:86 , srv1161952646:87 , srv122521629:87 , srv1471147673:87 , srv166912145:87 , srv1793093060:87 , srv193170916:87 , srv1976753015:87 , srv1981561157:87 , srv2074832508:87 , srv2091361836:87 , srv281233062:87 , srv286075388:87 , srv341857780:87 , srv530437413:87 , srv1046572669:88 , srv1128754426:88 , srv132975609:88 , srv1537325140:88 , srv189062829:88 , srv1961196575:88 , srv2147363000:88 , srv248521588:88 , srv597206712:88 , srv733015498:88 , srv884492301:88 , srv952860107:88 , srv97394816:88 , srv1139596000:89 , srv1403099424:89 , srv1425186077:89 , srv1560274941:89 , srv1819199192:89 , srv2045698210:89 , srv335020279:89 , srv670273939:89 , srv746862247:89 , srv947386842:89 , srv72333393:90 , srv897372335:90 , srv1063934431:91 , srv1173710940:91 , srv123503810:91 , srv1261439483:91 , srv1341974060:91 , srv1575680459:91 , srv1592106342:91 , srv1057095530:93 , srv271126469:96 , srv1862632179:441 } 2023-08-05 06:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1128754426=7, srv335020279=62, srv1967118206=43, srv132975609=16, srv1471147673=23, srv1633168401=28, srv1118667847=6, srv1793093060=31, srv1046572669=1, srv593544474=72, srv1359265432=18, srv2106026262=52, srv1261439483=15, srv530437413=71, srv72333393=79, srv1341974060=17, srv947386842=93, srv399483279=65, srv952860107=94, srv916236091=89, srv977639761=98, srv1173710940=11, srv631925823=75, srv166912145=29, srv525487263=70, srv769982771=82, srv2045698210=47, srv1139596000=8, srv1862632179=33, srv442883428=66, srv884492301=85, srv894739427=86, srv1864015269=34, srv50395223=69, srv1592106342=27, srv733015498=80, srv123503810=13, srv1976753015=44, srv838726459=84, srv2046124697=48, srv122521629=12, srv1144448281=9, srv1537325140=24, srv189062829=37, srv480530053=67, srv1057672839=3, srv1036315820=0, srv97394816=97, srv1819199192=32, srv1057095530=2, srv1728060904=30, srv1063934431=4, srv1380763891=20, srv281233062=58, srv286075388=60, srv1926511940=40, srv601286307=74, srv660658865=76, srv1920405275=39, srv193170916=41, srv902000959=88, srv923546395=91, srv670273939=77, srv67520564=78, srv942938409=92, srv978900089=99, srv1560274941=25, srv2147363000=53, srv314858622=61, srv341857780=63, srv800107385=83, srv897372335=87, srv1892953847=38, srv2074832508=49, srv2007567041=46, srv969959278=96, srv388873011=64, srv188373002=36, srv2077445258=50, srv746862247=81, srv255962218=56, srv1961196575=42, srv1095460410=5, srv28548298=59, srv597206712=73, srv1403099424=21, srv271126469=57, srv1575680459=26, srv1161952646=10, srv1425186077=22, srv919737266=90, srv1364803629=19, srv248521588=54, srv2091361836=51, srv1870833286=35, srv1981561157=45, srv500348082=68, srv125206130=14, srv255544890=55, srv96905327=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:12,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:12,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:12,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:12,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:12,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:12,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 06:53:12,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 06:53:12,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 06:53:12,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 06:53:12,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 06:53:12,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 06:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 06:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 06:53:12,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 06:53:12,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 06:53:12,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 06:53:12,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 06:53:12,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 06:53:12,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 06:53:12,905 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0395135024064937, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05648117789839482, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0395135024064937, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16975, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.237, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:12,905 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 06:53:12,906 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17460302594359473, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0395135024064937, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05648117789839482, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0395135024064937, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16975, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.237, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 06:53:13,300 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 521 ms to try 5000 different iterations. Found a solution that moves 2625 regions; Going from a computed imbalance of 0.17460302594359473 to a new imbalance of 0.01379401265628067. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020280487342608962); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033611632853451145, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020280487342608962); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:13,310 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1057095530:77 , srv271126469:77 , srv1261439483:79 , srv1063934431:81 , srv1128754426:81 , srv1139596000:81 , srv123503810:81 , srv2045698210:81 , srv897372335:81 , srv1425186077:82 , srv1575680459:82 , srv1592106342:82 , srv1819199192:82 , srv189062829:82 , srv1161952646:83 , srv1341974060:83 , srv1560274941:83 , srv193170916:83 , srv1961196575:83 , srv2074832508:83 , srv746862247:83 , srv1173710940:84 , srv1633168401:84 , srv1864015269:84 , srv2147363000:84 , srv248521588:84 , srv28548298:84 , srv335020279:84 , srv399483279:84 , srv597206712:84 , srv947386842:84 , srv97394816:84 , srv1364803629:85 , srv1403099424:85 , srv1537325140:85 , srv1976753015:85 , srv1981561157:85 , srv2007567041:85 , srv341857780:85 , srv442883428:85 , srv733015498:85 , srv884492301:85 , srv916236091:85 , srv952860107:85 , srv96905327:85 , srv1046572669:86 , srv1144448281:86 , srv122521629:86 , srv132975609:86 , srv1359265432:86 , srv1471147673:86 , srv1793093060:86 , srv1892953847:86 , srv2106026262:86 , srv255544890:86 , srv281233062:86 , srv50395223:86 , srv530437413:86 , srv601286307:86 , srv72333393:86 , srv919737266:86 , srv942938409:86 , srv977639761:86 , srv1036315820:87 , srv1095460410:87 , srv166912145:87 , srv1870833286:87 , srv1967118206:87 , srv2046124697:87 , srv314858622:87 , srv500348082:87 , srv670273939:87 , srv923546395:87 , srv1380763891:88 , srv2091361836:88 , srv255962218:88 , srv286075388:88 , srv631925823:88 , srv769982771:88 , srv838726459:88 , srv978900089:88 , srv1057672839:89 , srv125206130:89 , srv1920405275:89 , srv388873011:89 , srv593544474:89 , srv902000959:89 , srv969959278:89 , srv1728060904:90 , srv1926511940:90 , srv2077445258:90 , srv660658865:90 , srv894739427:90 , srv1118667847:91 , srv525487263:91 , srv67520564:91 , srv480530053:93 , srv800107385:93 , srv188373002:94 , srv1862632179:513 } 2023-08-05 06:53:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1128754426=7, srv335020279=62, srv1967118206=43, srv132975609=16, srv1471147673=23, srv1633168401=28, srv1118667847=6, srv1793093060=31, srv1046572669=1, srv593544474=72, srv1359265432=18, srv2106026262=52, srv1261439483=15, srv530437413=71, srv72333393=79, srv1341974060=17, srv947386842=93, srv399483279=65, srv952860107=94, srv916236091=89, srv977639761=98, srv1173710940=11, srv631925823=75, srv166912145=29, srv525487263=70, srv769982771=82, srv2045698210=47, srv1139596000=8, srv1862632179=33, srv442883428=66, srv884492301=85, srv894739427=86, srv1864015269=34, srv50395223=69, srv1592106342=27, srv733015498=80, srv123503810=13, srv1976753015=44, srv838726459=84, srv2046124697=48, srv122521629=12, srv1144448281=9, srv1537325140=24, srv189062829=37, srv480530053=67, srv1057672839=3, srv1036315820=0, srv97394816=97, srv1819199192=32, srv1057095530=2, srv1728060904=30, srv1063934431=4, srv1380763891=20, srv281233062=58, srv286075388=60, srv1926511940=40, srv601286307=74, srv660658865=76, srv1920405275=39, srv193170916=41, srv902000959=88, srv923546395=91, srv670273939=77, srv67520564=78, srv942938409=92, srv978900089=99, srv1560274941=25, srv2147363000=53, srv314858622=61, srv341857780=63, srv800107385=83, srv897372335=87, srv1892953847=38, srv2074832508=49, srv2007567041=46, srv969959278=96, srv388873011=64, srv188373002=36, srv2077445258=50, srv746862247=81, srv255962218=56, srv1961196575=42, srv1095460410=5, srv28548298=59, srv597206712=73, srv1403099424=21, srv271126469=57, srv1575680459=26, srv1161952646=10, srv1425186077=22, srv919737266=90, srv1364803629=19, srv248521588=54, srv2091361836=51, srv1870833286=35, srv1981561157=45, srv500348082=68, srv125206130=14, srv255544890=55, srv96905327=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:13,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:13,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:13,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:13,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 06:53:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 06:53:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 06:53:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 06:53:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 06:53:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 06:53:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 06:53:13,476 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020280487342608962); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033611632853451145, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020280487342608962); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:13,476 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 06:53:13,476 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013775632013423314, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020280487342608962); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033611632853451145, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020280487342608962); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 06:53:13,777 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations. Found a solution that moves 944 regions; Going from a computed imbalance of 0.013775632013423314 to a new imbalance of 6.516104600612038E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006668536961742503); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009487424484261084); MoveCostFunction : (multiplier=7.0, imbalance=0.10488888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006668536961742503); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:13,779 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv670273939:81 , srv72333393:81 , srv1173710940:82 , srv188373002:82 , srv286075388:82 , srv388873011:82 , srv1046572669:83 , srv1118667847:83 , srv125206130:83 , srv1341974060:83 , srv1380763891:83 , srv1728060904:83 , srv2091361836:83 , srv525487263:83 , srv1057672839:84 , srv132975609:84 , srv1403099424:84 , srv1471147673:84 , srv1592106342:84 , srv166912145:84 , srv1926511940:84 , srv271126469:84 , srv593544474:84 , srv601286307:84 , srv67520564:84 , srv733015498:84 , srv769982771:84 , srv884492301:84 , srv902000959:84 , srv942938409:84 , srv947386842:84 , srv969959278:84 , srv1063934431:85 , srv1144448281:85 , srv122521629:85 , srv123503810:85 , srv1537325140:85 , srv1560274941:85 , srv1793093060:85 , srv1981561157:85 , srv2147363000:85 , srv255962218:85 , srv281233062:85 , srv314858622:85 , srv335020279:85 , srv480530053:85 , srv500348082:85 , srv50395223:85 , srv530437413:85 , srv660658865:85 , srv746862247:85 , srv894739427:85 , srv919737266:85 , srv923546395:85 , srv952860107:85 , srv1057095530:86 , srv1095460410:86 , srv1359265432:86 , srv1425186077:86 , srv1575680459:86 , srv1819199192:86 , srv1870833286:86 , srv189062829:86 , srv1920405275:86 , srv1961196575:86 , srv1976753015:86 , srv2046124697:86 , srv248521588:86 , srv255544890:86 , srv341857780:86 , srv399483279:86 , srv597206712:86 , srv631925823:86 , srv800107385:86 , srv96905327:86 , srv97394816:86 , srv978900089:86 , srv1139596000:87 , srv1364803629:87 , srv193170916:87 , srv1967118206:87 , srv2074832508:87 , srv2077445258:87 , srv2106026262:87 , srv28548298:87 , srv442883428:87 , srv838726459:87 , srv897372335:87 , srv977639761:87 , srv1036315820:88 , srv1128754426:88 , srv1161952646:88 , srv1261439483:88 , srv1633168401:88 , srv1864015269:88 , srv1892953847:88 , srv2007567041:88 , srv2045698210:88 , srv916236091:88 , srv1862632179:563 } 2023-08-05 06:53:13,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:13,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1128754426=7, srv335020279=62, srv1967118206=43, srv132975609=16, srv1471147673=23, srv1633168401=28, srv1118667847=6, srv1793093060=31, srv1046572669=1, srv593544474=72, srv1359265432=18, srv2106026262=52, srv1261439483=15, srv530437413=71, srv72333393=79, srv1341974060=17, srv947386842=93, srv399483279=65, srv952860107=94, srv916236091=89, srv977639761=98, srv1173710940=11, srv631925823=75, srv166912145=29, srv525487263=70, srv769982771=82, srv2045698210=47, srv1139596000=8, srv1862632179=33, srv442883428=66, srv884492301=85, srv894739427=86, srv1864015269=34, srv50395223=69, srv1592106342=27, srv733015498=80, srv123503810=13, srv1976753015=44, srv838726459=84, srv2046124697=48, srv122521629=12, srv1144448281=9, srv1537325140=24, srv189062829=37, srv480530053=67, srv1057672839=3, srv1036315820=0, srv97394816=97, srv1819199192=32, srv1057095530=2, srv1728060904=30, srv1063934431=4, srv1380763891=20, srv281233062=58, srv286075388=60, srv1926511940=40, srv601286307=74, srv660658865=76, srv1920405275=39, srv193170916=41, srv902000959=88, srv923546395=91, srv670273939=77, srv67520564=78, srv942938409=92, srv978900089=99, srv1560274941=25, srv2147363000=53, srv314858622=61, srv341857780=63, srv800107385=83, srv897372335=87, srv1892953847=38, srv2074832508=49, srv2007567041=46, srv969959278=96, srv388873011=64, srv188373002=36, srv2077445258=50, srv746862247=81, srv255962218=56, srv1961196575=42, srv1095460410=5, srv28548298=59, srv597206712=73, srv1403099424=21, srv271126469=57, srv1575680459=26, srv1161952646=10, srv1425186077=22, srv919737266=90, srv1364803629=19, srv248521588=54, srv2091361836=51, srv1870833286=35, srv1981561157=45, srv500348082=68, srv125206130=14, srv255544890=55, srv96905327=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 06:53:13,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 06:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 06:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 06:53:13,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 06:53:13,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 06:53:13,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 06:53:13,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 06:53:13,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 06:53:13,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 06:53:13,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 06:53:13,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 06:53:13,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 06:53:13,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 06:53:13,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006668536961742503); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009487424484261084); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006668536961742503); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:13,944 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 06:53:13,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.450004307822153E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006668536961742503); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009487424484261084); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006668536961742503); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 06:53:14,404 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 585 ms to try 5000 different iterations. Found a solution that moves 285 regions; Going from a computed imbalance of 6.450004307822153E-4 to a new imbalance of 2.8819065412159403E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.03166666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:14,405 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv800107385:82 , srv1036315820:83 , srv1095460410:83 , srv1575680459:83 , srv1870833286:83 , srv916236091:83 , srv1144448281:84 , srv1161952646:84 , srv122521629:84 , srv123503810:84 , srv1261439483:84 , srv132975609:84 , srv1403099424:84 , srv1537325140:84 , srv1728060904:84 , srv1793093060:84 , srv1892953847:84 , srv1920405275:84 , srv1926511940:84 , srv193170916:84 , srv1976753015:84 , srv1981561157:84 , srv2007567041:84 , srv2045698210:84 , srv2046124697:84 , srv2074832508:84 , srv2077445258:84 , srv2147363000:84 , srv248521588:84 , srv255962218:84 , srv271126469:84 , srv281233062:84 , srv335020279:84 , srv480530053:84 , srv530437413:84 , srv597206712:84 , srv631925823:84 , srv660658865:84 , srv884492301:84 , srv897372335:84 , srv947386842:84 , srv952860107:84 , srv97394816:84 , srv977639761:84 , srv1046572669:85 , srv1057672839:85 , srv1063934431:85 , srv1128754426:85 , srv1139596000:85 , srv1173710940:85 , srv125206130:85 , srv1341974060:85 , srv1359265432:85 , srv1364803629:85 , srv1425186077:85 , srv1560274941:85 , srv1592106342:85 , srv1633168401:85 , srv166912145:85 , srv1819199192:85 , srv1864015269:85 , srv188373002:85 , srv1961196575:85 , srv1967118206:85 , srv2091361836:85 , srv2106026262:85 , srv255544890:85 , srv28548298:85 , srv286075388:85 , srv314858622:85 , srv341857780:85 , srv388873011:85 , srv442883428:85 , srv500348082:85 , srv50395223:85 , srv525487263:85 , srv593544474:85 , srv670273939:85 , srv72333393:85 , srv733015498:85 , srv746862247:85 , srv769982771:85 , srv838726459:85 , srv902000959:85 , srv923546395:85 , srv96905327:85 , srv969959278:85 , srv978900089:85 , srv1057095530:86 , srv1118667847:86 , srv1380763891:86 , srv1471147673:86 , srv189062829:86 , srv399483279:86 , srv601286307:86 , srv67520564:86 , srv894739427:86 , srv919737266:86 , srv942938409:86 , srv1862632179:625 } 2023-08-05 06:53:14,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:14,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1128754426=7, srv335020279=62, srv1967118206=43, srv132975609=16, srv1471147673=23, srv1633168401=28, srv1118667847=6, srv1793093060=31, srv1046572669=1, srv593544474=72, srv1359265432=18, srv2106026262=52, srv1261439483=15, srv530437413=71, srv72333393=79, srv1341974060=17, srv947386842=93, srv399483279=65, srv952860107=94, srv916236091=89, srv977639761=98, srv1173710940=11, srv631925823=75, srv166912145=29, srv525487263=70, srv769982771=82, srv2045698210=47, srv1139596000=8, srv1862632179=33, srv442883428=66, srv884492301=85, srv894739427=86, srv1864015269=34, srv50395223=69, srv1592106342=27, srv733015498=80, srv123503810=13, srv1976753015=44, srv838726459=84, srv2046124697=48, srv122521629=12, srv1144448281=9, srv1537325140=24, srv189062829=37, srv480530053=67, srv1057672839=3, srv1036315820=0, srv97394816=97, srv1819199192=32, srv1057095530=2, srv1728060904=30, srv1063934431=4, srv1380763891=20, srv281233062=58, srv286075388=60, srv1926511940=40, srv601286307=74, srv660658865=76, srv1920405275=39, srv193170916=41, srv902000959=88, srv923546395=91, srv670273939=77, srv67520564=78, srv942938409=92, srv978900089=99, srv1560274941=25, srv2147363000=53, srv314858622=61, srv341857780=63, srv800107385=83, srv897372335=87, srv1892953847=38, srv2074832508=49, srv2007567041=46, srv969959278=96, srv388873011=64, srv188373002=36, srv2077445258=50, srv746862247=81, srv255962218=56, srv1961196575=42, srv1095460410=5, srv28548298=59, srv597206712=73, srv1403099424=21, srv271126469=57, srv1575680459=26, srv1161952646=10, srv1425186077=22, srv919737266=90, srv1364803629=19, srv248521588=54, srv2091361836=51, srv1870833286=35, srv1981561157=45, srv500348082=68, srv125206130=14, srv255544890=55, srv96905327=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 06:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 06:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 06:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 06:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 06:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 06:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 06:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 06:53:14,563 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:14,563 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-05 06:53:14,564 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6823452759074905E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-05 06:53:14,850 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 403 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 2.6823452759074905E-5 to a new imbalance of 7.583817690791035E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:14,850 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1057672839:83 , srv1118667847:83 , srv1364803629:83 , srv1380763891:83 , srv1471147673:83 , srv166912145:83 , srv2091361836:83 , srv341857780:83 , srv67520564:83 , srv894739427:83 , srv923546395:83 , srv1036315820:84 , srv1046572669:84 , srv1057095530:84 , srv1063934431:84 , srv1128754426:84 , srv1139596000:84 , srv1161952646:84 , srv1173710940:84 , srv122521629:84 , srv125206130:84 , srv1261439483:84 , srv132975609:84 , srv1341974060:84 , srv1359265432:84 , srv1403099424:84 , srv1425186077:84 , srv1537325140:84 , srv1560274941:84 , srv1575680459:84 , srv1592106342:84 , srv1633168401:84 , srv1793093060:84 , srv1819199192:84 , srv1864015269:84 , srv1870833286:84 , srv188373002:84 , srv189062829:84 , srv1892953847:84 , srv1920405275:84 , srv1926511940:84 , srv1961196575:84 , srv1967118206:84 , srv1976753015:84 , srv2007567041:84 , srv2045698210:84 , srv2046124697:84 , srv2077445258:84 , srv2106026262:84 , srv248521588:84 , srv255544890:84 , srv255962218:84 , srv281233062:84 , srv28548298:84 , srv286075388:84 , srv314858622:84 , srv335020279:84 , srv388873011:84 , srv399483279:84 , srv442883428:84 , srv480530053:84 , srv500348082:84 , srv50395223:84 , srv525487263:84 , srv530437413:84 , srv593544474:84 , srv597206712:84 , srv601286307:84 , srv631925823:84 , srv660658865:84 , srv670273939:84 , srv72333393:84 , srv733015498:84 , srv746862247:84 , srv769982771:84 , srv800107385:84 , srv838726459:84 , srv897372335:84 , srv902000959:84 , srv916236091:84 , srv919737266:84 , srv942938409:84 , srv952860107:84 , srv96905327:84 , srv969959278:84 , srv97394816:84 , srv977639761:84 , srv978900089:84 , srv1095460410:85 , srv1144448281:85 , srv123503810:85 , srv1728060904:85 , srv193170916:85 , srv1981561157:85 , srv2074832508:85 , srv2147363000:85 , srv271126469:85 , srv884492301:85 , srv947386842:85 , srv1862632179:684 } 2023-08-05 06:53:14,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:14,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1128754426=7, srv335020279=62, srv1967118206=43, srv132975609=16, srv1471147673=23, srv1633168401=28, srv1118667847=6, srv1793093060=31, srv1046572669=1, srv593544474=72, srv1359265432=18, srv2106026262=52, srv1261439483=15, srv530437413=71, srv72333393=79, srv1341974060=17, srv947386842=93, srv399483279=65, srv952860107=94, srv916236091=89, srv977639761=98, srv1173710940=11, srv631925823=75, srv166912145=29, srv525487263=70, srv769982771=82, srv2045698210=47, srv1139596000=8, srv1862632179=33, srv442883428=66, srv884492301=85, srv894739427=86, srv1864015269=34, srv50395223=69, srv1592106342=27, srv733015498=80, srv123503810=13, srv1976753015=44, srv838726459=84, srv2046124697=48, srv122521629=12, srv1144448281=9, srv1537325140=24, srv189062829=37, srv480530053=67, srv1057672839=3, srv1036315820=0, srv97394816=97, srv1819199192=32, srv1057095530=2, srv1728060904=30, srv1063934431=4, srv1380763891=20, srv281233062=58, srv286075388=60, srv1926511940=40, srv601286307=74, srv660658865=76, srv1920405275=39, srv193170916=41, srv902000959=88, srv923546395=91, srv670273939=77, srv67520564=78, srv942938409=92, srv978900089=99, srv1560274941=25, srv2147363000=53, srv314858622=61, srv341857780=63, srv800107385=83, srv897372335=87, srv1892953847=38, srv2074832508=49, srv2007567041=46, srv969959278=96, srv388873011=64, srv188373002=36, srv2077445258=50, srv746862247=81, srv255962218=56, srv1961196575=42, srv1095460410=5, srv28548298=59, srv597206712=73, srv1403099424=21, srv271126469=57, srv1575680459=26, srv1161952646=10, srv1425186077=22, srv919737266=90, srv1364803629=19, srv248521588=54, srv2091361836=51, srv1870833286=35, srv1981561157=45, srv500348082=68, srv125206130=14, srv255544890=55, srv96905327=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-05 06:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-05 06:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-05 06:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-05 06:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-05 06:53:14,999 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:15,000 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1057672839:83 , srv1118667847:83 , srv1364803629:83 , srv1380763891:83 , srv1471147673:83 , srv166912145:83 , srv2091361836:83 , srv341857780:83 , srv67520564:83 , srv894739427:83 , srv923546395:83 , srv1036315820:84 , srv1046572669:84 , srv1057095530:84 , srv1063934431:84 , srv1128754426:84 , srv1139596000:84 , srv1161952646:84 , srv1173710940:84 , srv122521629:84 , srv125206130:84 , srv1261439483:84 , srv132975609:84 , srv1341974060:84 , srv1359265432:84 , srv1403099424:84 , srv1425186077:84 , srv1537325140:84 , srv1560274941:84 , srv1575680459:84 , srv1592106342:84 , srv1633168401:84 , srv1793093060:84 , srv1819199192:84 , srv1864015269:84 , srv1870833286:84 , srv188373002:84 , srv189062829:84 , srv1892953847:84 , srv1920405275:84 , srv1926511940:84 , srv1961196575:84 , srv1967118206:84 , srv1976753015:84 , srv2007567041:84 , srv2045698210:84 , srv2046124697:84 , srv2077445258:84 , srv2106026262:84 , srv248521588:84 , srv255544890:84 , srv255962218:84 , srv281233062:84 , srv28548298:84 , srv286075388:84 , srv314858622:84 , srv335020279:84 , srv388873011:84 , srv399483279:84 , srv442883428:84 , srv480530053:84 , srv500348082:84 , srv50395223:84 , srv525487263:84 , srv530437413:84 , srv593544474:84 , srv597206712:84 , srv601286307:84 , srv631925823:84 , srv660658865:84 , srv670273939:84 , srv72333393:84 , srv733015498:84 , srv746862247:84 , srv769982771:84 , srv800107385:84 , srv838726459:84 , srv897372335:84 , srv902000959:84 , srv916236091:84 , srv919737266:84 , srv942938409:84 , srv952860107:84 , srv96905327:84 , srv969959278:84 , srv97394816:84 , srv977639761:84 , srv978900089:84 , srv1095460410:85 , srv1144448281:85 , srv123503810:85 , srv1728060904:85 , srv193170916:85 , srv1981561157:85 , srv2074832508:85 , srv2147363000:85 , srv271126469:85 , srv884492301:85 , srv947386842:85 , srv1862632179:684 } 2023-08-05 06:53:15,041 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 305) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=1233 (was 2067) 2023-08-05 06:53:15,054 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=173, AvailableMemoryMB=1232 2023-08-05 06:53:15,055 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-05 06:53:15,056 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-05 06:53:15,070 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009995970:84 , srv1047766369:84 , srv110661298:84 , srv1186475395:84 , srv1208635894:84 , srv1209664098:84 , srv123203899:84 , srv1248932533:84 , srv1300792928:84 , srv1365873669:84 , srv1545525516:84 , srv1551952413:84 , srv1678162538:84 , srv1688451270:84 , srv1782377593:84 , srv1800586669:84 , srv1872769889:84 , srv1899989077:84 , srv1900375417:84 , srv1993759838:84 , srv2040477102:84 , srv2107419889:84 , srv338458262:84 , srv56911074:84 , srv570292216:84 , srv578679574:84 , srv645407778:84 , srv717390878:84 , srv748193089:84 , srv694591486:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-05 06:53:15,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:53:15,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1678162538=12, srv1300792928=8, srv2040477102=20, srv1248932533=7, srv645407778=26, srv748193089=29, srv2107419889=21, srv717390878=28, srv56911074=23, srv1993759838=19, srv578679574=25, srv1365873669=9, srv1872769889=16, srv1899989077=17, srv1047766369=1, srv123203899=6, srv570292216=24, srv338458262=22, srv1551952413=11, srv1186475395=3, srv110661298=2, srv1208635894=4, srv1688451270=13, srv1800586669=15, srv1545525516=10, srv1782377593=14, srv1209664098=5, srv1009995970=0, srv1900375417=18, srv694591486=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:53:15,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:53:15,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-05 06:53:15,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0765559529956128, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:53:15,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912434268572389, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0765559529956128, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-05 06:55:15,095 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5531716 different iterations. Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912434268572389 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:55:15,102 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1009995970:90 , srv1047766369:90 , srv110661298:90 , srv1186475395:90 , srv1208635894:90 , srv1209664098:90 , srv123203899:90 , srv1248932533:90 , srv1300792928:90 , srv1365873669:90 , srv1545525516:90 , srv1551952413:90 , srv1678162538:90 , srv1688451270:90 , srv1782377593:90 , srv1800586669:90 , srv1872769889:90 , srv1899989077:90 , srv1900375417:90 , srv1993759838:90 , srv2040477102:90 , srv2107419889:90 , srv338458262:90 , srv56911074:90 , srv570292216:90 , srv578679574:90 , srv645407778:90 , srv694591486:90 , srv717390878:90 , srv748193089:90 } 2023-08-05 06:55:15,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-05 06:55:15,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1678162538=12, srv1300792928=8, srv2040477102=20, srv1248932533=7, srv645407778=26, srv748193089=29, srv2107419889=21, srv717390878=28, srv56911074=23, srv1993759838=19, srv578679574=25, srv1365873669=9, srv1872769889=16, srv1899989077=17, srv1047766369=1, srv123203899=6, srv570292216=24, srv338458262=22, srv1551952413=11, srv1186475395=3, srv110661298=2, srv1208635894=4, srv1688451270=13, srv1800586669=15, srv1545525516=10, srv1782377593=14, srv1209664098=5, srv1009995970=0, srv1900375417=18, srv694591486=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-05 06:55:15,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-05 06:55:15,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-05 06:55:15,122 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-05 06:55:15,122 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1009995970:90 , srv1047766369:90 , srv110661298:90 , srv1186475395:90 , srv1208635894:90 , srv1209664098:90 , srv123203899:90 , srv1248932533:90 , srv1300792928:90 , srv1365873669:90 , srv1545525516:90 , srv1551952413:90 , srv1678162538:90 , srv1688451270:90 , srv1782377593:90 , srv1800586669:90 , srv1872769889:90 , srv1899989077:90 , srv1900375417:90 , srv1993759838:90 , srv2040477102:90 , srv2107419889:90 , srv338458262:90 , srv56911074:90 , srv570292216:90 , srv578679574:90 , srv645407778:90 , srv694591486:90 , srv717390878:90 , srv748193089:90 } 2023-08-05 06:55:15,135 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=164 (was 337), ProcessCount=171 (was 173), AvailableMemoryMB=1162 (was 1232)