2023-08-13 22:53:11,177 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-13 22:53:11,486 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-13 22:53:11,604 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-13 22:53:11,674 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-13 22:53:11,683 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-13 22:53:11,697 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=286, ProcessCount=173, AvailableMemoryMB=16463 2023-08-13 22:53:11,699 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-13 22:53:11,700 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-13 22:53:11,846 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000111334:84 , srv1054278199:84 , srv1062995879:84 , srv1071897535:84 , srv1100405317:84 , srv1117661921:84 , srv1121626815:84 , srv1150121050:84 , srv1189673274:84 , srv1195294038:84 , srv1198004428:84 , srv121029911:84 , srv1212799448:84 , srv1236051381:84 , srv1253302361:84 , srv1254929904:84 , srv1261479063:84 , srv1271875591:84 , srv1274394215:84 , srv1276349249:84 , srv1280186929:84 , srv1284163311:84 , srv1284960478:84 , srv1304056146:84 , srv1356431103:84 , srv139043603:84 , srv140205862:84 , srv1432487241:84 , srv144751419:84 , srv1449042643:84 , srv1480036475:84 , srv1482002441:84 , srv1493954064:84 , srv1497423478:84 , srv1503593339:84 , srv1505003716:84 , srv1512448685:84 , srv1519012905:84 , srv1524385188:84 , srv1529450167:84 , srv1567385983:84 , srv1567762551:84 , srv1580037816:84 , srv159787085:84 , srv1619764960:84 , srv1641555149:84 , srv1739521210:84 , srv1745471159:84 , srv1772350215:84 , srv1790412934:84 , srv1812838771:84 , srv1820309473:84 , srv1821290681:84 , srv1857536578:84 , srv1866483291:84 , srv1868944510:84 , srv1907619998:84 , srv1935848170:84 , srv1974657789:84 , srv2002461858:84 , srv2015362760:84 , srv2015815427:84 , srv2019419608:84 , srv2033923824:84 , srv203719125:84 , srv2037473366:84 , srv2076385874:84 , srv2076849520:84 , srv2077217516:84 , srv2099600159:84 , srv2116080155:84 , srv226898837:84 , srv274493298:84 , srv294263780:84 , srv311148049:84 , srv345586826:84 , srv358883515:84 , srv444334384:84 , srv469970486:84 , srv522112605:84 , srv541967565:84 , srv564321700:84 , srv572453180:84 , srv584103883:84 , srv60469442:84 , srv808344256:84 , srv812525587:84 , srv82655766:84 , srv846952747:84 , srv851487288:84 , srv889094316:84 , srv889779868:84 , srv899739181:84 , srv906228467:84 , srv918657916:84 , srv936618477:84 , srv939130140:84 , srv949307952:84 , srv991714032:84 , srv25384586:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-13 22:53:11,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:11,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253302361=14, srv274493298=73, srv1189673274=8, srv1271875591=17, srv2002461858=59, srv1505003716=35, srv2019419608=62, srv2076385874=66, srv1524385188=38, srv2077217516=68, srv906228467=94, srv1254929904=15, srv1482002441=31, srv1567385983=40, srv1857536578=53, srv564321700=82, srv121029911=11, srv159787085=43, srv1866483291=54, srv939130140=97, srv1519012905=37, srv139043603=25, srv2037473366=65, srv949307952=98, srv1580037816=42, srv1280186929=20, srv936618477=96, srv1054278199=1, srv469970486=79, srv851487288=90, srv2015362760=60, srv572453180=83, srv1198004428=10, srv541967565=81, srv358883515=77, srv1974657789=58, srv2099600159=69, srv1480036475=30, srv1820309473=51, srv1117661921=5, srv2076849520=67, srv444334384=78, srv226898837=71, srv1100405317=4, srv1071897535=3, srv1529450167=39, srv1000111334=0, srv1935848170=57, srv991714032=99, srv1432487241=27, srv1745471159=47, srv1150121050=7, srv1772350215=48, srv1276349249=19, srv203719125=64, srv1212799448=12, srv899739181=93, srv918657916=95, srv2015815427=61, srv889779868=92, srv808344256=86, srv1812838771=50, srv345586826=76, srv1821290681=52, srv1304056146=23, srv1567762551=41, srv2033923824=63, srv294263780=74, srv311148049=75, srv1236051381=13, srv1356431103=24, srv522112605=80, srv1274394215=18, srv1619764960=44, srv1062995879=2, srv25384586=72, srv82655766=88, srv846952747=89, srv584103883=84, srv1449042643=29, srv1493954064=32, srv1503593339=34, srv2116080155=70, srv1261479063=16, srv1790412934=49, srv140205862=26, srv1868944510=55, srv1512448685=36, srv1739521210=46, srv812525587=87, srv1121626815=6, srv1907619998=56, srv889094316=91, srv1284960478=22, srv144751419=28, srv1641555149=45, srv60469442=85, srv1284163311=21, srv1497423478=33, srv1195294038=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-13 22:53:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-13 22:53:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-13 22:53:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-13 22:53:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-13 22:53:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-13 22:53:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-13 22:53:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-13 22:53:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-13 22:53:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-13 22:53:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-13 22:53:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-13 22:53:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-13 22:53:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-13 22:53:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-13 22:53:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-13 22:53:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-13 22:53:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-13 22:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-13 22:53:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-13 22:53:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-13 22:53:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-13 22:53:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-13 22:53:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-13 22:53:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-13 22:53:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-13 22:53:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-13 22:53:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-13 22:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-13 22:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-13 22:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-13 22:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-13 22:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-13 22:53:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-13 22:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-13 22:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-13 22:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-13 22:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-13 22:53:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-13 22:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-13 22:53:11,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-13 22:53:12,201 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:12,202 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-13 22:53:12,202 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-13 22:53:12,591 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 568 ms to try 5000 different iterations. Found a solution that moves 4660 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18508984504844733. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041410774273893225, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056891019530807926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041410774273893225, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18083333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:12,606 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv140205862:81 , srv121029911:82 , srv1212799448:82 , srv808344256:82 , srv812525587:82 , srv1121626815:83 , srv1150121050:83 , srv1271875591:83 , srv1276349249:83 , srv1284163311:83 , srv939130140:83 , srv1071897535:84 , srv1274394215:84 , srv1304056146:84 , srv1356431103:84 , srv1449042643:84 , srv1480036475:84 , srv1482002441:84 , srv1497423478:84 , srv159787085:84 , srv1812838771:84 , srv1907619998:84 , srv2033923824:84 , srv2076849520:84 , srv311148049:84 , srv60469442:84 , srv846952747:84 , srv851487288:84 , srv899739181:84 , srv1054278199:85 , srv1253302361:85 , srv1580037816:85 , srv1820309473:85 , srv1935848170:85 , srv2015815427:85 , srv2099600159:85 , srv2116080155:85 , srv226898837:85 , srv274493298:85 , srv294263780:85 , srv358883515:85 , srv469970486:85 , srv564321700:85 , srv991714032:85 , srv1062995879:86 , srv1100405317:86 , srv1195294038:86 , srv1567385983:86 , srv1790412934:86 , srv1974657789:86 , srv203719125:86 , srv2076385874:86 , srv1117661921:87 , srv1189673274:87 , srv1236051381:87 , srv1261479063:87 , srv1432487241:87 , srv144751419:87 , srv1493954064:87 , srv1503593339:87 , srv1505003716:87 , srv1512448685:87 , srv1519012905:87 , srv1567762551:87 , srv1857536578:87 , srv1866483291:87 , srv1868944510:87 , srv2015362760:87 , srv2037473366:87 , srv345586826:87 , srv444334384:87 , srv541967565:87 , srv82655766:87 , srv889779868:87 , srv906228467:87 , srv918657916:87 , srv1772350215:88 , srv522112605:88 , srv572453180:88 , srv584103883:88 , srv889094316:88 , srv1198004428:89 , srv1524385188:89 , srv1739521210:89 , srv1821290681:89 , srv2002461858:89 , srv2077217516:89 , srv949307952:89 , srv1284960478:90 , srv1619764960:90 , srv1641555149:90 , srv1280186929:91 , srv139043603:91 , srv1529450167:91 , srv1254929904:92 , srv1745471159:92 , srv1000111334:93 , srv936618477:93 , srv2019419608:96 , srv25384586:458 } 2023-08-13 22:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253302361=14, srv274493298=73, srv1189673274=8, srv1271875591=17, srv2002461858=59, srv1505003716=35, srv2019419608=62, srv2076385874=66, srv1524385188=38, srv2077217516=68, srv906228467=94, srv1254929904=15, srv1482002441=31, srv1567385983=40, srv1857536578=53, srv564321700=82, srv121029911=11, srv159787085=43, srv1866483291=54, srv939130140=97, srv1519012905=37, srv139043603=25, srv2037473366=65, srv949307952=98, srv1580037816=42, srv1280186929=20, srv936618477=96, srv1054278199=1, srv469970486=79, srv851487288=90, srv2015362760=60, srv572453180=83, srv1198004428=10, srv541967565=81, srv358883515=77, srv1974657789=58, srv2099600159=69, srv1480036475=30, srv1820309473=51, srv1117661921=5, srv2076849520=67, srv444334384=78, srv226898837=71, srv1100405317=4, srv1071897535=3, srv1529450167=39, srv1000111334=0, srv1935848170=57, srv991714032=99, srv1432487241=27, srv1745471159=47, srv1150121050=7, srv1772350215=48, srv1276349249=19, srv203719125=64, srv1212799448=12, srv899739181=93, srv918657916=95, srv2015815427=61, srv889779868=92, srv808344256=86, srv1812838771=50, srv345586826=76, srv1821290681=52, srv1304056146=23, srv1567762551=41, srv2033923824=63, srv294263780=74, srv311148049=75, srv1236051381=13, srv1356431103=24, srv522112605=80, srv1274394215=18, srv1619764960=44, srv1062995879=2, srv25384586=72, srv82655766=88, srv846952747=89, srv584103883=84, srv1449042643=29, srv1493954064=32, srv1503593339=34, srv2116080155=70, srv1261479063=16, srv1790412934=49, srv140205862=26, srv1868944510=55, srv1512448685=36, srv1739521210=46, srv812525587=87, srv1121626815=6, srv1907619998=56, srv889094316=91, srv1284960478=22, srv144751419=28, srv1641555149=45, srv60469442=85, srv1284163311=21, srv1497423478=33, srv1195294038=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:12,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:12,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:12,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-13 22:53:12,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-13 22:53:12,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-13 22:53:12,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-13 22:53:12,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-13 22:53:12,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-13 22:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-13 22:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-13 22:53:12,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-13 22:53:12,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-13 22:53:12,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-13 22:53:12,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-13 22:53:12,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-13 22:53:12,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-13 22:53:12,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-13 22:53:12,822 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041410774273893225, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056891019530807926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041410774273893225, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18083333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:12,822 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-13 22:53:12,823 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18505721503103198, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041410774273893225, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056891019530807926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041410774273893225, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18083333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-13 22:53:13,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 496 ms to try 5000 different iterations. Found a solution that moves 2688 regions; Going from a computed imbalance of 0.18505721503103198 to a new imbalance of 0.01673742853536904. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022165471868651813); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0337216439469164, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2986666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022165471868651813); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:13,183 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2019419608:75 , srv1280186929:79 , srv936618477:79 , srv1000111334:80 , srv1529450167:80 , srv1745471159:80 , srv1284960478:81 , srv1524385188:81 , srv1198004428:82 , srv1254929904:82 , srv139043603:82 , srv1619764960:82 , srv1641555149:82 , srv2002461858:82 , srv1189673274:83 , srv1503593339:83 , srv1512448685:83 , srv1739521210:83 , srv2015362760:83 , srv444334384:83 , srv572453180:83 , srv889094316:83 , srv949307952:83 , srv1117661921:84 , srv1236051381:84 , srv345586826:84 , srv584103883:84 , srv906228467:84 , srv1100405317:85 , srv1195294038:85 , srv144751419:85 , srv1519012905:85 , srv1772350215:85 , srv1821290681:85 , srv1857536578:85 , srv1935848170:85 , srv2077217516:85 , srv358883515:85 , srv522112605:85 , srv541967565:85 , srv889779868:85 , srv1062995879:86 , srv1261479063:86 , srv1274394215:86 , srv1505003716:86 , srv1567385983:86 , srv1567762551:86 , srv1812838771:86 , srv1820309473:86 , srv1868944510:86 , srv1974657789:86 , srv203719125:86 , srv2076385874:86 , srv226898837:86 , srv82655766:86 , srv918657916:86 , srv939130140:86 , srv1054278199:87 , srv1253302361:87 , srv1432487241:87 , srv1480036475:87 , srv1482002441:87 , srv1493954064:87 , srv1790412934:87 , srv1866483291:87 , srv2015815427:87 , srv2037473366:87 , srv2116080155:87 , srv274493298:87 , srv294263780:87 , srv846952747:87 , srv1071897535:88 , srv1150121050:88 , srv1212799448:88 , srv1284163311:88 , srv1356431103:88 , srv159787085:88 , srv2033923824:88 , srv2076849520:88 , srv2099600159:88 , srv469970486:88 , srv564321700:88 , srv991714032:88 , srv1121626815:89 , srv1271875591:89 , srv1304056146:89 , srv140205862:89 , srv1449042643:89 , srv1497423478:89 , srv808344256:89 , srv812525587:89 , srv851487288:89 , srv121029911:90 , srv1580037816:90 , srv899739181:90 , srv1276349249:91 , srv1907619998:91 , srv60469442:92 , srv311148049:93 , srv25384586:513 } 2023-08-13 22:53:13,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:13,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253302361=14, srv274493298=73, srv1189673274=8, srv1271875591=17, srv2002461858=59, srv1505003716=35, srv2019419608=62, srv2076385874=66, srv1524385188=38, srv2077217516=68, srv906228467=94, srv1254929904=15, srv1482002441=31, srv1567385983=40, srv1857536578=53, srv564321700=82, srv121029911=11, srv159787085=43, srv1866483291=54, srv939130140=97, srv1519012905=37, srv139043603=25, srv2037473366=65, srv949307952=98, srv1580037816=42, srv1280186929=20, srv936618477=96, srv1054278199=1, srv469970486=79, srv851487288=90, srv2015362760=60, srv572453180=83, srv1198004428=10, srv541967565=81, srv358883515=77, srv1974657789=58, srv2099600159=69, srv1480036475=30, srv1820309473=51, srv1117661921=5, srv2076849520=67, srv444334384=78, srv226898837=71, srv1100405317=4, srv1071897535=3, srv1529450167=39, srv1000111334=0, srv1935848170=57, srv991714032=99, srv1432487241=27, srv1745471159=47, srv1150121050=7, srv1772350215=48, srv1276349249=19, srv203719125=64, srv1212799448=12, srv899739181=93, srv918657916=95, srv2015815427=61, srv889779868=92, srv808344256=86, srv1812838771=50, srv345586826=76, srv1821290681=52, srv1304056146=23, srv1567762551=41, srv2033923824=63, srv294263780=74, srv311148049=75, srv1236051381=13, srv1356431103=24, srv522112605=80, srv1274394215=18, srv1619764960=44, srv1062995879=2, srv25384586=72, srv82655766=88, srv846952747=89, srv584103883=84, srv1449042643=29, srv1493954064=32, srv1503593339=34, srv2116080155=70, srv1261479063=16, srv1790412934=49, srv140205862=26, srv1868944510=55, srv1512448685=36, srv1739521210=46, srv812525587=87, srv1121626815=6, srv1907619998=56, srv889094316=91, srv1284960478=22, srv144751419=28, srv1641555149=45, srv60469442=85, srv1284163311=21, srv1497423478=33, srv1195294038=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-13 22:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-13 22:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-13 22:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-13 22:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-13 22:53:13,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-13 22:53:13,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-13 22:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-13 22:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-13 22:53:13,368 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022165471868651813); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0337216439469164, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022165471868651813); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:13,368 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-13 22:53:13,368 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016718606757083105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022165471868651813); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0337216439469164, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022165471868651813); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-13 22:53:13,602 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 340 ms to try 5000 different iterations. Found a solution that moves 1014 regions; Going from a computed imbalance of 0.016718606757083105 to a new imbalance of 5.616751607399928E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.11266666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:13,605 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv311148049:80 , srv1580037816:82 , srv1907619998:82 , srv60469442:82 , srv1254929904:83 , srv1276349249:83 , srv1432487241:83 , srv1493954064:83 , srv1000111334:84 , srv1261479063:84 , srv1304056146:84 , srv1505003716:84 , srv1567762551:84 , srv1772350215:84 , srv1821290681:84 , srv1866483291:84 , srv1868944510:84 , srv2037473366:84 , srv2077217516:84 , srv2099600159:84 , srv469970486:84 , srv522112605:84 , srv899739181:84 , srv991714032:84 , srv1054278199:85 , srv1071897535:85 , srv1121626815:85 , srv121029911:85 , srv1253302361:85 , srv139043603:85 , srv1449042643:85 , srv1497423478:85 , srv1567385983:85 , srv1619764960:85 , srv1641555149:85 , srv1745471159:85 , srv1790412934:85 , srv1974657789:85 , srv2015815427:85 , srv2033923824:85 , srv203719125:85 , srv2076385874:85 , srv2076849520:85 , srv564321700:85 , srv808344256:85 , srv82655766:85 , srv851487288:85 , srv918657916:85 , srv1062995879:86 , srv1117661921:86 , srv1212799448:86 , srv1271875591:86 , srv144751419:86 , srv1519012905:86 , srv159787085:86 , srv1739521210:86 , srv1820309473:86 , srv2019419608:86 , srv2116080155:86 , srv226898837:86 , srv274493298:86 , srv345586826:86 , srv358883515:86 , srv541967565:86 , srv584103883:86 , srv812525587:86 , srv889094316:86 , srv889779868:86 , srv906228467:86 , srv936618477:86 , srv949307952:86 , srv1100405317:87 , srv1150121050:87 , srv1189673274:87 , srv1195294038:87 , srv1198004428:87 , srv1236051381:87 , srv1280186929:87 , srv1284163311:87 , srv1284960478:87 , srv1356431103:87 , srv140205862:87 , srv1480036475:87 , srv1482002441:87 , srv1503593339:87 , srv1512448685:87 , srv1524385188:87 , srv1529450167:87 , srv1812838771:87 , srv1857536578:87 , srv2002461858:87 , srv2015362760:87 , srv294263780:87 , srv572453180:87 , srv846952747:87 , srv1274394215:88 , srv1935848170:88 , srv444334384:88 , srv939130140:89 , srv25384586:539 } 2023-08-13 22:53:13,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:13,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253302361=14, srv274493298=73, srv1189673274=8, srv1271875591=17, srv2002461858=59, srv1505003716=35, srv2019419608=62, srv2076385874=66, srv1524385188=38, srv2077217516=68, srv906228467=94, srv1254929904=15, srv1482002441=31, srv1567385983=40, srv1857536578=53, srv564321700=82, srv121029911=11, srv159787085=43, srv1866483291=54, srv939130140=97, srv1519012905=37, srv139043603=25, srv2037473366=65, srv949307952=98, srv1580037816=42, srv1280186929=20, srv936618477=96, srv1054278199=1, srv469970486=79, srv851487288=90, srv2015362760=60, srv572453180=83, srv1198004428=10, srv541967565=81, srv358883515=77, srv1974657789=58, srv2099600159=69, srv1480036475=30, srv1820309473=51, srv1117661921=5, srv2076849520=67, srv444334384=78, srv226898837=71, srv1100405317=4, srv1071897535=3, srv1529450167=39, srv1000111334=0, srv1935848170=57, srv991714032=99, srv1432487241=27, srv1745471159=47, srv1150121050=7, srv1772350215=48, srv1276349249=19, srv203719125=64, srv1212799448=12, srv899739181=93, srv918657916=95, srv2015815427=61, srv889779868=92, srv808344256=86, srv1812838771=50, srv345586826=76, srv1821290681=52, srv1304056146=23, srv1567762551=41, srv2033923824=63, srv294263780=74, srv311148049=75, srv1236051381=13, srv1356431103=24, srv522112605=80, srv1274394215=18, srv1619764960=44, srv1062995879=2, srv25384586=72, srv82655766=88, srv846952747=89, srv584103883=84, srv1449042643=29, srv1493954064=32, srv1503593339=34, srv2116080155=70, srv1261479063=16, srv1790412934=49, srv140205862=26, srv1868944510=55, srv1512448685=36, srv1739521210=46, srv812525587=87, srv1121626815=6, srv1907619998=56, srv889094316=91, srv1284960478=22, srv144751419=28, srv1641555149=45, srv60469442=85, srv1284163311=21, srv1497423478=33, srv1195294038=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-13 22:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-13 22:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-13 22:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-13 22:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-13 22:53:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-13 22:53:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-13 22:53:13,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-13 22:53:13,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-13 22:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-13 22:53:13,778 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:13,778 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-13 22:53:13,778 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.545749809848079E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005875203800993194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005875203800993194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-13 22:53:14,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 512 ms to try 5000 different iterations. Found a solution that moves 256 regions; Going from a computed imbalance of 5.545749809848079E-4 to a new imbalance of 5.036467362565269E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.028444444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:14,171 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1356431103:82 , srv1100405317:83 , srv1857536578:83 , srv294263780:83 , srv444334384:83 , srv1062995879:84 , srv1150121050:84 , srv1195294038:84 , srv1236051381:84 , srv1261479063:84 , srv1271875591:84 , srv1274394215:84 , srv1284163311:84 , srv1284960478:84 , srv139043603:84 , srv144751419:84 , srv1449042643:84 , srv1480036475:84 , srv1497423478:84 , srv1519012905:84 , srv1524385188:84 , srv1529450167:84 , srv159787085:84 , srv1739521210:84 , srv1790412934:84 , srv1821290681:84 , srv1866483291:84 , srv1935848170:84 , srv2002461858:84 , srv2015362760:84 , srv203719125:84 , srv2037473366:84 , srv2077217516:84 , srv2116080155:84 , srv274493298:84 , srv541967565:84 , srv564321700:84 , srv572453180:84 , srv584103883:84 , srv82655766:84 , srv846952747:84 , srv851487288:84 , srv899739181:84 , srv918657916:84 , srv936618477:84 , srv949307952:84 , srv1000111334:85 , srv1054278199:85 , srv1071897535:85 , srv1117661921:85 , srv1121626815:85 , srv1189673274:85 , srv1198004428:85 , srv121029911:85 , srv1253302361:85 , srv1254929904:85 , srv1276349249:85 , srv1280186929:85 , srv1304056146:85 , srv140205862:85 , srv1482002441:85 , srv1493954064:85 , srv1503593339:85 , srv1505003716:85 , srv1512448685:85 , srv1567385983:85 , srv1567762551:85 , srv1619764960:85 , srv1641555149:85 , srv1745471159:85 , srv1772350215:85 , srv1812838771:85 , srv1820309473:85 , srv1868944510:85 , srv1907619998:85 , srv1974657789:85 , srv2015815427:85 , srv2019419608:85 , srv2033923824:85 , srv2076385874:85 , srv2076849520:85 , srv2099600159:85 , srv226898837:85 , srv311148049:85 , srv345586826:85 , srv469970486:85 , srv522112605:85 , srv60469442:85 , srv812525587:85 , srv889094316:85 , srv889779868:85 , srv906228467:85 , srv939130140:85 , srv991714032:85 , srv1212799448:86 , srv1432487241:86 , srv1580037816:86 , srv358883515:86 , srv808344256:86 , srv25384586:632 } 2023-08-13 22:53:14,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:14,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253302361=14, srv274493298=73, srv1189673274=8, srv1271875591=17, srv2002461858=59, srv1505003716=35, srv2019419608=62, srv2076385874=66, srv1524385188=38, srv2077217516=68, srv906228467=94, srv1254929904=15, srv1482002441=31, srv1567385983=40, srv1857536578=53, srv564321700=82, srv121029911=11, srv159787085=43, srv1866483291=54, srv939130140=97, srv1519012905=37, srv139043603=25, srv2037473366=65, srv949307952=98, srv1580037816=42, srv1280186929=20, srv936618477=96, srv1054278199=1, srv469970486=79, srv851487288=90, srv2015362760=60, srv572453180=83, srv1198004428=10, srv541967565=81, srv358883515=77, srv1974657789=58, srv2099600159=69, srv1480036475=30, srv1820309473=51, srv1117661921=5, srv2076849520=67, srv444334384=78, srv226898837=71, srv1100405317=4, srv1071897535=3, srv1529450167=39, srv1000111334=0, srv1935848170=57, srv991714032=99, srv1432487241=27, srv1745471159=47, srv1150121050=7, srv1772350215=48, srv1276349249=19, srv203719125=64, srv1212799448=12, srv899739181=93, srv918657916=95, srv2015815427=61, srv889779868=92, srv808344256=86, srv1812838771=50, srv345586826=76, srv1821290681=52, srv1304056146=23, srv1567762551=41, srv2033923824=63, srv294263780=74, srv311148049=75, srv1236051381=13, srv1356431103=24, srv522112605=80, srv1274394215=18, srv1619764960=44, srv1062995879=2, srv25384586=72, srv82655766=88, srv846952747=89, srv584103883=84, srv1449042643=29, srv1493954064=32, srv1503593339=34, srv2116080155=70, srv1261479063=16, srv1790412934=49, srv140205862=26, srv1868944510=55, srv1512448685=36, srv1739521210=46, srv812525587=87, srv1121626815=6, srv1907619998=56, srv889094316=91, srv1284960478=22, srv144751419=28, srv1641555149=45, srv60469442=85, srv1284163311=21, srv1497423478=33, srv1195294038=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:14,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:14,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-13 22:53:14,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-13 22:53:14,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-13 22:53:14,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-13 22:53:14,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-13 22:53:14,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-13 22:53:14,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-13 22:53:14,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-13 22:53:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-13 22:53:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-13 22:53:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-13 22:53:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-13 22:53:14,309 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:14,310 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-13 22:53:14,310 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.857212331270661E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-13 22:53:14,582 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 375 ms to try 5000 different iterations. Found a solution that moves 27 regions; Going from a computed imbalance of 4.857212331270661E-5 to a new imbalance of 8.16874815785912E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.003); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:14,583 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1198004428:83 , srv1432487241:83 , srv1482002441:83 , srv1580037816:83 , srv60469442:83 , srv889779868:83 , srv939130140:83 , srv1000111334:84 , srv1054278199:84 , srv1062995879:84 , srv1071897535:84 , srv1117661921:84 , srv1121626815:84 , srv1150121050:84 , srv1189673274:84 , srv1195294038:84 , srv121029911:84 , srv1212799448:84 , srv1236051381:84 , srv1253302361:84 , srv1254929904:84 , srv1271875591:84 , srv1274394215:84 , srv1276349249:84 , srv1280186929:84 , srv1284163311:84 , srv1284960478:84 , srv1304056146:84 , srv139043603:84 , srv140205862:84 , srv144751419:84 , srv1449042643:84 , srv1480036475:84 , srv1493954064:84 , srv1497423478:84 , srv1503593339:84 , srv1505003716:84 , srv1512448685:84 , srv1519012905:84 , srv1529450167:84 , srv1567385983:84 , srv1567762551:84 , srv159787085:84 , srv1619764960:84 , srv1641555149:84 , srv1739521210:84 , srv1745471159:84 , srv1772350215:84 , srv1790412934:84 , srv1812838771:84 , srv1820309473:84 , srv1821290681:84 , srv1857536578:84 , srv1866483291:84 , srv1868944510:84 , srv1907619998:84 , srv1935848170:84 , srv1974657789:84 , srv2002461858:84 , srv2015815427:84 , srv2019419608:84 , srv2033923824:84 , srv2037473366:84 , srv2076385874:84 , srv2076849520:84 , srv2077217516:84 , srv2099600159:84 , srv2116080155:84 , srv226898837:84 , srv274493298:84 , srv294263780:84 , srv311148049:84 , srv345586826:84 , srv358883515:84 , srv469970486:84 , srv522112605:84 , srv541967565:84 , srv564321700:84 , srv572453180:84 , srv584103883:84 , srv808344256:84 , srv812525587:84 , srv82655766:84 , srv846952747:84 , srv851487288:84 , srv889094316:84 , srv899739181:84 , srv906228467:84 , srv918657916:84 , srv936618477:84 , srv949307952:84 , srv991714032:84 , srv1100405317:85 , srv1261479063:85 , srv1356431103:85 , srv1524385188:85 , srv2015362760:85 , srv203719125:85 , srv444334384:85 , srv25384586:684 } 2023-08-13 22:53:14,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:14,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253302361=14, srv274493298=73, srv1189673274=8, srv1271875591=17, srv2002461858=59, srv1505003716=35, srv2019419608=62, srv2076385874=66, srv1524385188=38, srv2077217516=68, srv906228467=94, srv1254929904=15, srv1482002441=31, srv1567385983=40, srv1857536578=53, srv564321700=82, srv121029911=11, srv159787085=43, srv1866483291=54, srv939130140=97, srv1519012905=37, srv139043603=25, srv2037473366=65, srv949307952=98, srv1580037816=42, srv1280186929=20, srv936618477=96, srv1054278199=1, srv469970486=79, srv851487288=90, srv2015362760=60, srv572453180=83, srv1198004428=10, srv541967565=81, srv358883515=77, srv1974657789=58, srv2099600159=69, srv1480036475=30, srv1820309473=51, srv1117661921=5, srv2076849520=67, srv444334384=78, srv226898837=71, srv1100405317=4, srv1071897535=3, srv1529450167=39, srv1000111334=0, srv1935848170=57, srv991714032=99, srv1432487241=27, srv1745471159=47, srv1150121050=7, srv1772350215=48, srv1276349249=19, srv203719125=64, srv1212799448=12, srv899739181=93, srv918657916=95, srv2015815427=61, srv889779868=92, srv808344256=86, srv1812838771=50, srv345586826=76, srv1821290681=52, srv1304056146=23, srv1567762551=41, srv2033923824=63, srv294263780=74, srv311148049=75, srv1236051381=13, srv1356431103=24, srv522112605=80, srv1274394215=18, srv1619764960=44, srv1062995879=2, srv25384586=72, srv82655766=88, srv846952747=89, srv584103883=84, srv1449042643=29, srv1493954064=32, srv1503593339=34, srv2116080155=70, srv1261479063=16, srv1790412934=49, srv140205862=26, srv1868944510=55, srv1512448685=36, srv1739521210=46, srv812525587=87, srv1121626815=6, srv1907619998=56, srv889094316=91, srv1284960478=22, srv144751419=28, srv1641555149=45, srv60469442=85, srv1284163311=21, srv1497423478=33, srv1195294038=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-13 22:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-13 22:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-13 22:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-13 22:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-13 22:53:14,729 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:14,729 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1198004428:83 , srv1432487241:83 , srv1482002441:83 , srv1580037816:83 , srv60469442:83 , srv889779868:83 , srv939130140:83 , srv1000111334:84 , srv1054278199:84 , srv1062995879:84 , srv1071897535:84 , srv1117661921:84 , srv1121626815:84 , srv1150121050:84 , srv1189673274:84 , srv1195294038:84 , srv121029911:84 , srv1212799448:84 , srv1236051381:84 , srv1253302361:84 , srv1254929904:84 , srv1271875591:84 , srv1274394215:84 , srv1276349249:84 , srv1280186929:84 , srv1284163311:84 , srv1284960478:84 , srv1304056146:84 , srv139043603:84 , srv140205862:84 , srv144751419:84 , srv1449042643:84 , srv1480036475:84 , srv1493954064:84 , srv1497423478:84 , srv1503593339:84 , srv1505003716:84 , srv1512448685:84 , srv1519012905:84 , srv1529450167:84 , srv1567385983:84 , srv1567762551:84 , srv159787085:84 , srv1619764960:84 , srv1641555149:84 , srv1739521210:84 , srv1745471159:84 , srv1772350215:84 , srv1790412934:84 , srv1812838771:84 , srv1820309473:84 , srv1821290681:84 , srv1857536578:84 , srv1866483291:84 , srv1868944510:84 , srv1907619998:84 , srv1935848170:84 , srv1974657789:84 , srv2002461858:84 , srv2015815427:84 , srv2019419608:84 , srv2033923824:84 , srv2037473366:84 , srv2076385874:84 , srv2076849520:84 , srv2077217516:84 , srv2099600159:84 , srv2116080155:84 , srv226898837:84 , srv274493298:84 , srv294263780:84 , srv311148049:84 , srv345586826:84 , srv358883515:84 , srv469970486:84 , srv522112605:84 , srv541967565:84 , srv564321700:84 , srv572453180:84 , srv584103883:84 , srv808344256:84 , srv812525587:84 , srv82655766:84 , srv846952747:84 , srv851487288:84 , srv889094316:84 , srv899739181:84 , srv906228467:84 , srv918657916:84 , srv936618477:84 , srv949307952:84 , srv991714032:84 , srv1100405317:85 , srv1261479063:85 , srv1356431103:85 , srv1524385188:85 , srv2015362760:85 , srv203719125:85 , srv444334384:85 , srv25384586:684 } 2023-08-13 22:53:14,759 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=286 (was 286), ProcessCount=173 (was 173), AvailableMemoryMB=15616 (was 16463) 2023-08-13 22:53:14,771 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=286, ProcessCount=173, AvailableMemoryMB=15616 2023-08-13 22:53:14,772 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-13 22:53:14,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-13 22:53:14,789 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1064567053:84 , srv1065769018:84 , srv1069498692:84 , srv1125058855:84 , srv1315618875:84 , srv1319196378:84 , srv1337897433:84 , srv1347631188:84 , srv1427058537:84 , srv1459346022:84 , srv1461061326:84 , srv1606482801:84 , srv1703242785:84 , srv1779766361:84 , srv1795440172:84 , srv1828863496:84 , srv1882367150:84 , srv1892804351:84 , srv205982236:84 , srv2068032165:84 , srv2082171819:84 , srv2087237666:84 , srv409924355:84 , srv531705517:84 , srv592642060:84 , srv709551792:84 , srv881148212:84 , srv907301015:84 , srv912203484:84 , srv1557173344:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-13 22:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1828863496=16, srv2068032165=20, srv1892804351=18, srv912203484=29, srv881148212=27, srv1337897433=6, srv1882367150=17, srv1319196378=5, srv205982236=19, srv2087237666=22, srv1315618875=4, srv1064567053=0, srv409924355=23, srv1461061326=10, srv1125058855=3, srv1779766361=14, srv907301015=28, srv1703242785=13, srv2082171819=21, srv1347631188=7, srv709551792=26, srv1459346022=9, srv1557173344=11, srv1065769018=1, srv1427058537=8, srv592642060=25, srv1069498692=2, srv1795440172=15, srv531705517=24, srv1606482801=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-13 22:53:14,836 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1490865606301943, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:53:14,836 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.99126628101487, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1490865606301943, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-13 22:55:14,811 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5936663 different iterations. Found a solution that moves 1857 regions; Going from a computed imbalance of 0.99126628101487 to a new imbalance of 4.334330639506329E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:55:14,818 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1064567053:90 , srv1065769018:90 , srv1069498692:90 , srv1125058855:90 , srv1315618875:90 , srv1319196378:90 , srv1337897433:90 , srv1347631188:90 , srv1427058537:90 , srv1459346022:90 , srv1461061326:90 , srv1557173344:90 , srv1606482801:90 , srv1703242785:90 , srv1779766361:90 , srv1795440172:90 , srv1828863496:90 , srv1882367150:90 , srv1892804351:90 , srv205982236:90 , srv2068032165:90 , srv2082171819:90 , srv2087237666:90 , srv409924355:90 , srv531705517:90 , srv592642060:90 , srv709551792:90 , srv881148212:90 , srv907301015:90 , srv912203484:90 } 2023-08-13 22:55:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-13 22:55:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1828863496=16, srv2068032165=20, srv1892804351=18, srv912203484=29, srv881148212=27, srv1337897433=6, srv1882367150=17, srv1319196378=5, srv205982236=19, srv2087237666=22, srv1315618875=4, srv1064567053=0, srv409924355=23, srv1461061326=10, srv1125058855=3, srv1779766361=14, srv907301015=28, srv1703242785=13, srv2082171819=21, srv1347631188=7, srv709551792=26, srv1459346022=9, srv1557173344=11, srv1065769018=1, srv1427058537=8, srv592642060=25, srv1069498692=2, srv1795440172=15, srv531705517=24, srv1606482801=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-13 22:55:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-13 22:55:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-13 22:55:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-13 22:55:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-13 22:55:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-13 22:55:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-13 22:55:14,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-13 22:55:14,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-13 22:55:14,837 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-13 22:55:14,837 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1064567053:90 , srv1065769018:90 , srv1069498692:90 , srv1125058855:90 , srv1315618875:90 , srv1319196378:90 , srv1337897433:90 , srv1347631188:90 , srv1427058537:90 , srv1459346022:90 , srv1461061326:90 , srv1557173344:90 , srv1606482801:90 , srv1703242785:90 , srv1779766361:90 , srv1795440172:90 , srv1828863496:90 , srv1882367150:90 , srv1892804351:90 , srv205982236:90 , srv2068032165:90 , srv2082171819:90 , srv2087237666:90 , srv409924355:90 , srv531705517:90 , srv592642060:90 , srv709551792:90 , srv881148212:90 , srv907301015:90 , srv912203484:90 } 2023-08-13 22:55:14,858 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=192 (was 286), ProcessCount=173 (was 173), AvailableMemoryMB=15382 (was 15616)