2023-07-22 19:53:13,411 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-22 19:53:13,731 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-22 19:53:13,843 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-22 19:53:13,910 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-22 19:53:13,918 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-22 19:53:13,936 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=269, ProcessCount=175, AvailableMemoryMB=10990 2023-07-22 19:53:13,938 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-22 19:53:13,938 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-22 19:53:14,102 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003533909:84 , srv1031644554:84 , srv1087714138:84 , srv109331542:84 , srv1099409595:84 , srv1102111468:84 , srv1128608:84 , srv1131626082:84 , srv1145519234:84 , srv1163757654:84 , srv1167709861:84 , srv1171608570:84 , srv1176802781:84 , srv1183233643:84 , srv1187793697:84 , srv1240792788:84 , srv1246624748:84 , srv1272463991:84 , srv1276717011:84 , srv1280217693:84 , srv1296735611:84 , srv1304275463:84 , srv1313782561:84 , srv1319463466:84 , srv132794349:84 , srv1357355525:84 , srv1382963701:84 , srv1388813841:84 , srv1391111446:84 , srv1402589973:84 , srv1429222981:84 , srv1433474221:84 , srv1441277124:84 , srv1461991191:84 , srv1490271386:84 , srv1491045456:84 , srv1523540845:84 , srv1532700144:84 , srv1537780281:84 , srv1541990695:84 , srv1546564658:84 , srv1571963633:84 , srv1583394542:84 , srv161707419:84 , srv164467068:84 , srv1669290451:84 , srv1676022272:84 , srv1684999511:84 , srv1695102104:84 , srv1724991687:84 , srv1744021166:84 , srv1762737330:84 , srv180540291:84 , srv182946153:84 , srv1833517569:84 , srv1857539920:84 , srv1870514665:84 , srv1871331634:84 , srv1877977410:84 , srv1881758162:84 , srv1916727887:84 , srv1941427548:84 , srv2064931716:84 , srv2066203867:84 , srv2080951354:84 , srv2088895802:84 , srv2123034734:84 , srv255459511:84 , srv271341061:84 , srv318368727:84 , srv326315308:84 , srv409259970:84 , srv413543704:84 , srv477256898:84 , srv486314268:84 , srv567539341:84 , srv61675132:84 , srv640296055:84 , srv652275095:84 , srv653225969:84 , srv691677952:84 , srv691867707:84 , srv698698193:84 , srv736188249:84 , srv750219448:84 , srv766944349:84 , srv778378622:84 , srv786732611:84 , srv812880583:84 , srv85114593:84 , srv851573976:84 , srv858122258:84 , srv868246745:84 , srv873020700:84 , srv874894298:84 , srv888369291:84 , srv928139688:84 , srv939947574:84 , srv966331153:84 , srv435820878:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-22 19:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv812880583=89, srv750219448=85, srv1128608=6, srv1099409595=4, srv1532700144=37, srv1280217693=19, srv1003533909=0, srv1433474221=31, srv1183233643=13, srv409259970=71, srv786732611=88, srv255459511=67, srv691677952=81, srv2088895802=65, srv1833517569=54, srv1176802781=12, srv567539341=76, srv1087714138=2, srv1541990695=39, srv873020700=94, srv1163757654=9, srv1571963633=41, srv271341061=68, srv868246745=93, srv1102111468=5, srv1304275463=21, srv1724991687=49, srv1246624748=16, srv1546564658=40, srv1461991191=33, srv652275095=79, srv1296735611=20, srv1131626082=7, srv640296055=78, srv1523540845=36, srv2064931716=62, srv109331542=3, srv1669290451=45, srv1941427548=61, srv1537780281=38, srv1240792788=15, srv1762737330=51, srv1881758162=59, srv161707419=43, srv653225969=80, srv1382963701=26, srv778378622=87, srv61675132=77, srv928139688=97, srv1388813841=27, srv1272463991=17, srv1490271386=34, srv413543704=72, srv736188249=84, srv1676022272=46, srv1313782561=22, srv2080951354=64, srv851573976=91, srv1871331634=57, srv326315308=70, srv1695102104=48, srv1429222981=30, srv1031644554=1, srv1491045456=35, srv164467068=44, srv180540291=52, srv1857539920=55, srv435820878=73, srv486314268=75, srv477256898=74, srv1916727887=60, srv966331153=99, srv874894298=95, srv1877977410=58, srv1145519234=8, srv182946153=53, srv766944349=86, srv1583394542=42, srv698698193=83, srv888369291=96, srv939947574=98, srv1167709861=10, srv1319463466=23, srv1391111446=28, srv318368727=69, srv691867707=82, srv1744021166=50, srv1402589973=29, srv1684999511=47, srv132794349=24, srv1441277124=32, srv858122258=92, srv1171608570=11, srv1187793697=14, srv1870514665=56, srv2123034734=66, srv1276717011=18, srv1357355525=25, srv2066203867=63, srv85114593=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:53:14,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:53:14,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:53:14,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-22 19:53:14,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-22 19:53:14,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-22 19:53:14,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-22 19:53:14,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-22 19:53:14,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-22 19:53:14,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-22 19:53:14,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-22 19:53:14,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-22 19:53:14,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-22 19:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-22 19:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-22 19:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-22 19:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-22 19:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-22 19:53:14,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:14,421 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-22 19:53:14,421 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-22 19:53:14,803 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 539 ms to try 5000 different iterations. Found a solution that moves 4728 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17245612006803973. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04083918317809132, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05743097159474579, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5253333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04083918317809132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16733333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:14,822 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1490271386:81 , srv858122258:81 , srv1280217693:82 , srv132794349:82 , srv1429222981:82 , srv1877977410:82 , srv109331542:83 , srv1491045456:83 , srv1724991687:83 , srv182946153:83 , srv2080951354:83 , srv486314268:83 , srv1240792788:84 , srv1433474221:84 , srv1532700144:84 , srv1546564658:84 , srv1871331634:84 , srv1881758162:84 , srv2064931716:84 , srv326315308:84 , srv874894298:84 , srv1099409595:85 , srv1167709861:85 , srv1171608570:85 , srv1296735611:85 , srv1319463466:85 , srv1388813841:85 , srv1391111446:85 , srv1402589973:85 , srv164467068:85 , srv1684999511:85 , srv1744021166:85 , srv271341061:85 , srv477256898:85 , srv691677952:85 , srv691867707:85 , srv750219448:85 , srv939947574:85 , srv1102111468:86 , srv1145519234:86 , srv1176802781:86 , srv1357355525:86 , srv1441277124:86 , srv1523540845:86 , srv1762737330:86 , srv1916727887:86 , srv318368727:86 , srv409259970:86 , srv567539341:86 , srv652275095:86 , srv653225969:86 , srv851573976:86 , srv868246745:86 , srv928139688:86 , srv1003533909:87 , srv1128608:87 , srv1131626082:87 , srv1187793697:87 , srv1246624748:87 , srv1382963701:87 , srv1541990695:87 , srv1571963633:87 , srv1583394542:87 , srv1669290451:87 , srv1833517569:87 , srv1870514665:87 , srv2088895802:87 , srv413543704:87 , srv736188249:87 , srv766944349:87 , srv778378622:87 , srv786732611:87 , srv888369291:87 , srv1272463991:88 , srv1461991191:88 , srv1857539920:88 , srv1941427548:88 , srv2066203867:88 , srv255459511:88 , srv640296055:88 , srv698698193:88 , srv812880583:88 , srv873020700:88 , srv1031644554:89 , srv1304275463:89 , srv1313782561:89 , srv1183233643:90 , srv1276717011:90 , srv1537780281:90 , srv1676022272:90 , srv2123034734:90 , srv61675132:90 , srv85114593:90 , srv966331153:90 , srv1087714138:91 , srv1163757654:91 , srv161707419:91 , srv180540291:94 , srv1695102104:95 , srv435820878:453 } 2023-07-22 19:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv812880583=89, srv750219448=85, srv1128608=6, srv1099409595=4, srv1532700144=37, srv1280217693=19, srv1003533909=0, srv1433474221=31, srv1183233643=13, srv409259970=71, srv786732611=88, srv255459511=67, srv691677952=81, srv2088895802=65, srv1833517569=54, srv1176802781=12, srv567539341=76, srv1087714138=2, srv1541990695=39, srv873020700=94, srv1163757654=9, srv1571963633=41, srv271341061=68, srv868246745=93, srv1102111468=5, srv1304275463=21, srv1724991687=49, srv1246624748=16, srv1546564658=40, srv1461991191=33, srv652275095=79, srv1296735611=20, srv1131626082=7, srv640296055=78, srv1523540845=36, srv2064931716=62, srv109331542=3, srv1669290451=45, srv1941427548=61, srv1537780281=38, srv1240792788=15, srv1762737330=51, srv1881758162=59, srv161707419=43, srv653225969=80, srv1382963701=26, srv778378622=87, srv61675132=77, srv928139688=97, srv1388813841=27, srv1272463991=17, srv1490271386=34, srv413543704=72, srv736188249=84, srv1676022272=46, srv1313782561=22, srv2080951354=64, srv851573976=91, srv1871331634=57, srv326315308=70, srv1695102104=48, srv1429222981=30, srv1031644554=1, srv1491045456=35, srv164467068=44, srv180540291=52, srv1857539920=55, srv435820878=73, srv486314268=75, srv477256898=74, srv1916727887=60, srv966331153=99, srv874894298=95, srv1877977410=58, srv1145519234=8, srv182946153=53, srv766944349=86, srv1583394542=42, srv698698193=83, srv888369291=96, srv939947574=98, srv1167709861=10, srv1319463466=23, srv1391111446=28, srv318368727=69, srv691867707=82, srv1744021166=50, srv1402589973=29, srv1684999511=47, srv132794349=24, srv1441277124=32, srv858122258=92, srv1171608570=11, srv1187793697=14, srv1870514665=56, srv2123034734=66, srv1276717011=18, srv1357355525=25, srv2066203867=63, srv85114593=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:53:14,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:53:14,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:53:14,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:53:14,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:53:14,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:53:14,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:53:14,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:53:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:53:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:53:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:53:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:53:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:53:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:53:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:53:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-22 19:53:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-22 19:53:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-22 19:53:14,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-22 19:53:14,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-22 19:53:14,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-22 19:53:14,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-22 19:53:14,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-22 19:53:14,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-22 19:53:14,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-22 19:53:14,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-22 19:53:14,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-22 19:53:14,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-22 19:53:15,050 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04083918317809132, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05743097159474579, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04083918317809132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16733333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:15,051 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-22 19:53:15,051 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17242301390444753, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04083918317809132, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05743097159474579, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04083918317809132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16733333333333333, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-22 19:53:15,443 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 537 ms to try 5000 different iterations. Found a solution that moves 2669 regions; Going from a computed imbalance of 0.17242301390444753 to a new imbalance of 0.012682681808719662. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020596540570609); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02798508501264527, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29655555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020596540570609); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:15,455 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1695102104:77 , srv180540291:77 , srv161707419:79 , srv2123034734:80 , srv1537780281:81 , srv61675132:81 , srv1087714138:82 , srv1183233643:82 , srv1676022272:82 , srv698698193:82 , srv1031644554:83 , srv1272463991:83 , srv1304275463:83 , srv1313782561:83 , srv1461991191:83 , srv1833517569:83 , srv2066203867:83 , srv85114593:83 , srv966331153:83 , srv1163757654:84 , srv1176802781:84 , srv1357355525:84 , srv1762737330:84 , srv640296055:84 , srv1003533909:85 , srv1102111468:85 , srv1128608:85 , srv1131626082:85 , srv1187793697:85 , srv1276717011:85 , srv1391111446:85 , srv1571963633:85 , srv1669290451:85 , srv1857539920:85 , srv1941427548:85 , srv2088895802:85 , srv409259970:85 , srv413543704:85 , srv477256898:85 , srv653225969:85 , srv691867707:85 , srv786732611:85 , srv812880583:85 , srv873020700:85 , srv888369291:85 , srv1145519234:86 , srv1246624748:86 , srv1382963701:86 , srv1388813841:86 , srv1523540845:86 , srv1541990695:86 , srv1744021166:86 , srv1870514665:86 , srv2064931716:86 , srv271341061:86 , srv691677952:86 , srv736188249:86 , srv766944349:86 , srv778378622:86 , srv851573976:86 , srv1167709861:87 , srv1319463466:87 , srv1402589973:87 , srv1441277124:87 , srv1871331634:87 , srv1881758162:87 , srv1916727887:87 , srv255459511:87 , srv318368727:87 , srv326315308:87 , srv567539341:87 , srv652275095:87 , srv750219448:87 , srv874894298:87 , srv928139688:87 , srv109331542:88 , srv1240792788:88 , srv132794349:88 , srv1491045456:88 , srv1532700144:88 , srv1546564658:88 , srv1684999511:88 , srv868246745:88 , srv939947574:88 , srv1171608570:89 , srv1296735611:89 , srv1433474221:89 , srv1583394542:89 , srv164467068:89 , srv1099409595:90 , srv1280217693:90 , srv1429222981:90 , srv1724991687:90 , srv2080951354:90 , srv486314268:90 , srv1490271386:91 , srv858122258:91 , srv182946153:93 , srv1877977410:94 , srv435820878:504 } 2023-07-22 19:53:15,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:53:15,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv812880583=89, srv750219448=85, srv1128608=6, srv1099409595=4, srv1532700144=37, srv1280217693=19, srv1003533909=0, srv1433474221=31, srv1183233643=13, srv409259970=71, srv786732611=88, srv255459511=67, srv691677952=81, srv2088895802=65, srv1833517569=54, srv1176802781=12, srv567539341=76, srv1087714138=2, srv1541990695=39, srv873020700=94, srv1163757654=9, srv1571963633=41, srv271341061=68, srv868246745=93, srv1102111468=5, srv1304275463=21, srv1724991687=49, srv1246624748=16, srv1546564658=40, srv1461991191=33, srv652275095=79, srv1296735611=20, srv1131626082=7, srv640296055=78, srv1523540845=36, srv2064931716=62, srv109331542=3, srv1669290451=45, srv1941427548=61, srv1537780281=38, srv1240792788=15, srv1762737330=51, srv1881758162=59, srv161707419=43, srv653225969=80, srv1382963701=26, srv778378622=87, srv61675132=77, srv928139688=97, srv1388813841=27, srv1272463991=17, srv1490271386=34, srv413543704=72, srv736188249=84, srv1676022272=46, srv1313782561=22, srv2080951354=64, srv851573976=91, srv1871331634=57, srv326315308=70, srv1695102104=48, srv1429222981=30, srv1031644554=1, srv1491045456=35, srv164467068=44, srv180540291=52, srv1857539920=55, srv435820878=73, srv486314268=75, srv477256898=74, srv1916727887=60, srv966331153=99, srv874894298=95, srv1877977410=58, srv1145519234=8, srv182946153=53, srv766944349=86, srv1583394542=42, srv698698193=83, srv888369291=96, srv939947574=98, srv1167709861=10, srv1319463466=23, srv1391111446=28, srv318368727=69, srv691867707=82, srv1744021166=50, srv1402589973=29, srv1684999511=47, srv132794349=24, srv1441277124=32, srv858122258=92, srv1171608570=11, srv1187793697=14, srv1870514665=56, srv2123034734=66, srv1276717011=18, srv1357355525=25, srv2066203867=63, srv85114593=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:53:15,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:53:15,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:53:15,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-22 19:53:15,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-22 19:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-22 19:53:15,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-22 19:53:15,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-22 19:53:15,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-22 19:53:15,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-22 19:53:15,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-22 19:53:15,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-22 19:53:15,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-22 19:53:15,623 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020596540570609); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02798508501264527, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020596540570609); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:15,623 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-22 19:53:15,624 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01266399307127727, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020596540570609); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02798508501264527, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020596540570609); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-22 19:53:15,958 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 998 regions; Going from a computed imbalance of 0.01266399307127727 to a new imbalance of 4.668171499908484E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004655483366657976); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005504818825631802); MoveCostFunction : (multiplier=7.0, imbalance=0.11088888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004655483366657976); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:15,962 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1583394542:81 , srv182946153:81 , srv1099409595:82 , srv1877977410:82 , srv255459511:82 , srv1163757654:83 , srv1171608570:83 , srv1276717011:83 , srv1296735611:84 , srv1382963701:84 , srv1541990695:84 , srv1684999511:84 , srv1724991687:84 , srv652275095:84 , srv812880583:84 , srv85114593:84 , srv939947574:84 , srv966331153:84 , srv1087714138:85 , srv1145519234:85 , srv1167709861:85 , srv1183233643:85 , srv1240792788:85 , srv1246624748:85 , srv1280217693:85 , srv1402589973:85 , srv1429222981:85 , srv1433474221:85 , srv1441277124:85 , srv1490271386:85 , srv1523540845:85 , srv1532700144:85 , srv1571963633:85 , srv164467068:85 , srv1669290451:85 , srv1857539920:85 , srv1870514665:85 , srv1941427548:85 , srv2080951354:85 , srv2088895802:85 , srv318368727:85 , srv413543704:85 , srv486314268:85 , srv567539341:85 , srv736188249:85 , srv750219448:85 , srv766944349:85 , srv778378622:85 , srv858122258:85 , srv868246745:85 , srv873020700:85 , srv888369291:85 , srv928139688:85 , srv1003533909:86 , srv1031644554:86 , srv109331542:86 , srv1131626082:86 , srv1187793697:86 , srv1272463991:86 , srv1304275463:86 , srv1313782561:86 , srv1319463466:86 , srv1388813841:86 , srv1461991191:86 , srv1546564658:86 , srv1676022272:86 , srv1695102104:86 , srv1744021166:86 , srv180540291:86 , srv1916727887:86 , srv2066203867:86 , srv640296055:86 , srv698698193:86 , srv786732611:86 , srv851573976:86 , srv1128608:87 , srv1176802781:87 , srv1357355525:87 , srv1491045456:87 , srv1762737330:87 , srv1833517569:87 , srv1871331634:87 , srv1881758162:87 , srv2064931716:87 , srv2123034734:87 , srv271341061:87 , srv326315308:87 , srv409259970:87 , srv477256898:87 , srv61675132:87 , srv653225969:87 , srv691677952:87 , srv691867707:87 , srv874894298:87 , srv1102111468:88 , srv1391111446:88 , srv1537780281:88 , srv161707419:88 , srv132794349:89 , srv435820878:542 } 2023-07-22 19:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:53:15,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv812880583=89, srv750219448=85, srv1128608=6, srv1099409595=4, srv1532700144=37, srv1280217693=19, srv1003533909=0, srv1433474221=31, srv1183233643=13, srv409259970=71, srv786732611=88, srv255459511=67, srv691677952=81, srv2088895802=65, srv1833517569=54, srv1176802781=12, srv567539341=76, srv1087714138=2, srv1541990695=39, srv873020700=94, srv1163757654=9, srv1571963633=41, srv271341061=68, srv868246745=93, srv1102111468=5, srv1304275463=21, srv1724991687=49, srv1246624748=16, srv1546564658=40, srv1461991191=33, srv652275095=79, srv1296735611=20, srv1131626082=7, srv640296055=78, srv1523540845=36, srv2064931716=62, srv109331542=3, srv1669290451=45, srv1941427548=61, srv1537780281=38, srv1240792788=15, srv1762737330=51, srv1881758162=59, srv161707419=43, srv653225969=80, srv1382963701=26, srv778378622=87, srv61675132=77, srv928139688=97, srv1388813841=27, srv1272463991=17, srv1490271386=34, srv413543704=72, srv736188249=84, srv1676022272=46, srv1313782561=22, srv2080951354=64, srv851573976=91, srv1871331634=57, srv326315308=70, srv1695102104=48, srv1429222981=30, srv1031644554=1, srv1491045456=35, srv164467068=44, srv180540291=52, srv1857539920=55, srv435820878=73, srv486314268=75, srv477256898=74, srv1916727887=60, srv966331153=99, srv874894298=95, srv1877977410=58, srv1145519234=8, srv182946153=53, srv766944349=86, srv1583394542=42, srv698698193=83, srv888369291=96, srv939947574=98, srv1167709861=10, srv1319463466=23, srv1391111446=28, srv318368727=69, srv691867707=82, srv1744021166=50, srv1402589973=29, srv1684999511=47, srv132794349=24, srv1441277124=32, srv858122258=92, srv1171608570=11, srv1187793697=14, srv1870514665=56, srv2123034734=66, srv1276717011=18, srv1357355525=25, srv2066203867=63, srv85114593=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-22 19:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-22 19:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-22 19:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-22 19:53:16,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-22 19:53:16,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-22 19:53:16,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-22 19:53:16,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-22 19:53:16,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-22 19:53:16,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-22 19:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-22 19:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-22 19:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-22 19:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-22 19:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-22 19:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-22 19:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-22 19:53:16,174 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004655483366657976); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005504818825631802); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004655483366657976); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:16,174 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-22 19:53:16,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.598290046302228E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004655483366657976); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005504818825631802); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004655483366657976); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-22 19:53:16,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 512 ms to try 5000 different iterations. Found a solution that moves 208 regions; Going from a computed imbalance of 4.598290046302228E-4 to a new imbalance of 1.0811634648707658E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.02311111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:16,542 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1102111468:83 , srv1128608:83 , srv1163757654:83 , srv132794349:83 , srv1537780281:83 , srv164467068:83 , srv1916727887:83 , srv318368727:83 , srv868246745:83 , srv1003533909:84 , srv1031644554:84 , srv1087714138:84 , srv1131626082:84 , srv1167709861:84 , srv1187793697:84 , srv1246624748:84 , srv1272463991:84 , srv1276717011:84 , srv1296735611:84 , srv1304275463:84 , srv1313782561:84 , srv1391111446:84 , srv1433474221:84 , srv1441277124:84 , srv1532700144:84 , srv1546564658:84 , srv161707419:84 , srv1676022272:84 , srv1695102104:84 , srv180540291:84 , srv1857539920:84 , srv1870514665:84 , srv1871331634:84 , srv1877977410:84 , srv1881758162:84 , srv1941427548:84 , srv2064931716:84 , srv2080951354:84 , srv271341061:84 , srv326315308:84 , srv409259970:84 , srv567539341:84 , srv61675132:84 , srv640296055:84 , srv691677952:84 , srv736188249:84 , srv750219448:84 , srv766944349:84 , srv778378622:84 , srv786732611:84 , srv851573976:84 , srv873020700:84 , srv874894298:84 , srv928139688:84 , srv1099409595:85 , srv1145519234:85 , srv1171608570:85 , srv1176802781:85 , srv1183233643:85 , srv1280217693:85 , srv1319463466:85 , srv1357355525:85 , srv1388813841:85 , srv1402589973:85 , srv1461991191:85 , srv1490271386:85 , srv1491045456:85 , srv1523540845:85 , srv1541990695:85 , srv1571963633:85 , srv1583394542:85 , srv1669290451:85 , srv1684999511:85 , srv1724991687:85 , srv1744021166:85 , srv182946153:85 , srv1833517569:85 , srv2066203867:85 , srv2088895802:85 , srv2123034734:85 , srv255459511:85 , srv413543704:85 , srv477256898:85 , srv486314268:85 , srv652275095:85 , srv653225969:85 , srv691867707:85 , srv698698193:85 , srv812880583:85 , srv85114593:85 , srv858122258:85 , srv888369291:85 , srv939947574:85 , srv966331153:85 , srv109331542:86 , srv1240792788:86 , srv1382963701:86 , srv1429222981:86 , srv1762737330:86 , srv435820878:643 } 2023-07-22 19:53:16,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:53:16,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv812880583=89, srv750219448=85, srv1128608=6, srv1099409595=4, srv1532700144=37, srv1280217693=19, srv1003533909=0, srv1433474221=31, srv1183233643=13, srv409259970=71, srv786732611=88, srv255459511=67, srv691677952=81, srv2088895802=65, srv1833517569=54, srv1176802781=12, srv567539341=76, srv1087714138=2, srv1541990695=39, srv873020700=94, srv1163757654=9, srv1571963633=41, srv271341061=68, srv868246745=93, srv1102111468=5, srv1304275463=21, srv1724991687=49, srv1246624748=16, srv1546564658=40, srv1461991191=33, srv652275095=79, srv1296735611=20, srv1131626082=7, srv640296055=78, srv1523540845=36, srv2064931716=62, srv109331542=3, srv1669290451=45, srv1941427548=61, srv1537780281=38, srv1240792788=15, srv1762737330=51, srv1881758162=59, srv161707419=43, srv653225969=80, srv1382963701=26, srv778378622=87, srv61675132=77, srv928139688=97, srv1388813841=27, srv1272463991=17, srv1490271386=34, srv413543704=72, srv736188249=84, srv1676022272=46, srv1313782561=22, srv2080951354=64, srv851573976=91, srv1871331634=57, srv326315308=70, srv1695102104=48, srv1429222981=30, srv1031644554=1, srv1491045456=35, srv164467068=44, srv180540291=52, srv1857539920=55, srv435820878=73, srv486314268=75, srv477256898=74, srv1916727887=60, srv966331153=99, srv874894298=95, srv1877977410=58, srv1145519234=8, srv182946153=53, srv766944349=86, srv1583394542=42, srv698698193=83, srv888369291=96, srv939947574=98, srv1167709861=10, srv1319463466=23, srv1391111446=28, srv318368727=69, srv691867707=82, srv1744021166=50, srv1402589973=29, srv1684999511=47, srv132794349=24, srv1441277124=32, srv858122258=92, srv1171608570=11, srv1187793697=14, srv1870514665=56, srv2123034734=66, srv1276717011=18, srv1357355525=25, srv2066203867=63, srv85114593=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:53:16,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:53:16,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:53:16,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-22 19:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-22 19:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-22 19:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-22 19:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-22 19:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-22 19:53:16,688 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.355187519438973E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:16,688 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1102111468:83 , srv1128608:83 , srv1163757654:83 , srv132794349:83 , srv1537780281:83 , srv164467068:83 , srv1916727887:83 , srv318368727:83 , srv868246745:83 , srv1003533909:84 , srv1031644554:84 , srv1087714138:84 , srv1131626082:84 , srv1167709861:84 , srv1187793697:84 , srv1246624748:84 , srv1272463991:84 , srv1276717011:84 , srv1296735611:84 , srv1304275463:84 , srv1313782561:84 , srv1391111446:84 , srv1433474221:84 , srv1441277124:84 , srv1532700144:84 , srv1546564658:84 , srv161707419:84 , srv1676022272:84 , srv1695102104:84 , srv180540291:84 , srv1857539920:84 , srv1870514665:84 , srv1871331634:84 , srv1877977410:84 , srv1881758162:84 , srv1941427548:84 , srv2064931716:84 , srv2080951354:84 , srv271341061:84 , srv326315308:84 , srv409259970:84 , srv567539341:84 , srv61675132:84 , srv640296055:84 , srv691677952:84 , srv736188249:84 , srv750219448:84 , srv766944349:84 , srv778378622:84 , srv786732611:84 , srv851573976:84 , srv873020700:84 , srv874894298:84 , srv928139688:84 , srv1099409595:85 , srv1145519234:85 , srv1171608570:85 , srv1176802781:85 , srv1183233643:85 , srv1280217693:85 , srv1319463466:85 , srv1357355525:85 , srv1388813841:85 , srv1402589973:85 , srv1461991191:85 , srv1490271386:85 , srv1491045456:85 , srv1523540845:85 , srv1541990695:85 , srv1571963633:85 , srv1583394542:85 , srv1669290451:85 , srv1684999511:85 , srv1724991687:85 , srv1744021166:85 , srv182946153:85 , srv1833517569:85 , srv2066203867:85 , srv2088895802:85 , srv2123034734:85 , srv255459511:85 , srv413543704:85 , srv477256898:85 , srv486314268:85 , srv652275095:85 , srv653225969:85 , srv691867707:85 , srv698698193:85 , srv812880583:85 , srv85114593:85 , srv858122258:85 , srv888369291:85 , srv939947574:85 , srv966331153:85 , srv109331542:86 , srv1240792788:86 , srv1382963701:86 , srv1429222981:86 , srv1762737330:86 , srv435820878:643 } 2023-07-22 19:53:16,728 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 269) - SystemLoadAverage LEAK? -, ProcessCount=175 (was 175), AvailableMemoryMB=10180 (was 10990) 2023-07-22 19:53:16,741 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=175, AvailableMemoryMB=10179 2023-07-22 19:53:16,742 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-22 19:53:16,742 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-22 19:53:16,759 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1048525255:84 , srv1160846184:84 , srv1220406732:84 , srv1243802191:84 , srv1356809257:84 , srv1358244002:84 , srv1378368706:84 , srv140164126:84 , srv1441282255:84 , srv1486449334:84 , srv1514296684:84 , srv1622042783:84 , srv1693784420:84 , srv1722594499:84 , srv1839310234:84 , srv1928719588:84 , srv2094457783:84 , srv236204758:84 , srv29675189:84 , srv297027151:84 , srv336046222:84 , srv368323053:84 , srv375980156:84 , srv47773899:84 , srv72594281:84 , srv740698973:84 , srv812026465:84 , srv937193301:84 , srv962717857:84 , srv1082404390:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-22 19:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv740698973=26, srv1378368706=7, srv297027151=20, srv1514296684=11, srv812026465=27, srv1486449334=10, srv1622042783=12, srv1243802191=4, srv140164126=8, srv336046222=21, srv1082404390=1, srv1441282255=9, srv2094457783=17, srv1048525255=0, srv1839310234=15, srv1220406732=3, srv375980156=23, srv1356809257=5, srv1722594499=14, srv962717857=29, srv29675189=19, srv368323053=22, srv937193301=28, srv1358244002=6, srv72594281=25, srv47773899=24, srv1928719588=16, srv236204758=18, srv1160846184=2, srv1693784420=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:53:16,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:53:16,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:53:16,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:53:16,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-22 19:53:16,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.116786426933157, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:53:16,808 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912561033482119, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.116786426933157, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-22 19:55:16,783 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6079490 different iterations. Found a solution that moves 1863 regions; Going from a computed imbalance of 0.9912561033482119 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:55:16,788 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1048525255:90 , srv1082404390:90 , srv1160846184:90 , srv1220406732:90 , srv1243802191:90 , srv1356809257:90 , srv1358244002:90 , srv1378368706:90 , srv140164126:90 , srv1441282255:90 , srv1486449334:90 , srv1514296684:90 , srv1622042783:90 , srv1693784420:90 , srv1722594499:90 , srv1839310234:90 , srv1928719588:90 , srv2094457783:90 , srv236204758:90 , srv29675189:90 , srv297027151:90 , srv336046222:90 , srv368323053:90 , srv375980156:90 , srv47773899:90 , srv72594281:90 , srv740698973:90 , srv812026465:90 , srv937193301:90 , srv962717857:90 } 2023-07-22 19:55:16,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-22 19:55:16,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv740698973=26, srv1378368706=7, srv297027151=20, srv1514296684=11, srv812026465=27, srv1486449334=10, srv1622042783=12, srv1243802191=4, srv140164126=8, srv336046222=21, srv1082404390=1, srv1441282255=9, srv2094457783=17, srv1048525255=0, srv1839310234=15, srv1220406732=3, srv375980156=23, srv1356809257=5, srv1722594499=14, srv962717857=29, srv29675189=19, srv368323053=22, srv937193301=28, srv1358244002=6, srv72594281=25, srv47773899=24, srv1928719588=16, srv236204758=18, srv1160846184=2, srv1693784420=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-22 19:55:16,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-22 19:55:16,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-22 19:55:16,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-22 19:55:16,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-22 19:55:16,807 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1048525255:90 , srv1082404390:90 , srv1160846184:90 , srv1220406732:90 , srv1243802191:90 , srv1356809257:90 , srv1358244002:90 , srv1378368706:90 , srv140164126:90 , srv1441282255:90 , srv1486449334:90 , srv1514296684:90 , srv1622042783:90 , srv1693784420:90 , srv1722594499:90 , srv1839310234:90 , srv1928719588:90 , srv2094457783:90 , srv236204758:90 , srv29675189:90 , srv297027151:90 , srv336046222:90 , srv368323053:90 , srv375980156:90 , srv47773899:90 , srv72594281:90 , srv740698973:90 , srv812026465:90 , srv937193301:90 , srv962717857:90 } 2023-07-22 19:55:16,821 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=227 (was 296), ProcessCount=171 (was 175), AvailableMemoryMB=10066 (was 10179)