2023-06-01 15:53:25,082 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-06-01 15:53:25,377 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-06-01 15:53:25,490 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-06-01 15:53:25,549 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-01 15:53:25,557 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-01 15:53:25,570 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=269, ProcessCount=173, AvailableMemoryMB=2308 2023-06-01 15:53:25,572 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-01 15:53:25,572 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-01 15:53:25,696 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021057827:84 , srv1023280128:84 , srv1044490786:84 , srv106495684:84 , srv1090589006:84 , srv1092957260:84 , srv1102074384:84 , srv1175072163:84 , srv1176703863:84 , srv1180458308:84 , srv1212803779:84 , srv1225771019:84 , srv124038085:84 , srv124368945:84 , srv1250327139:84 , srv132078957:84 , srv1330872238:84 , srv1342808762:84 , srv1347998407:84 , srv1363645478:84 , srv1374428035:84 , srv1401838630:84 , srv1431219291:84 , srv1512797508:84 , srv15266221:84 , srv1537024694:84 , srv1569319342:84 , srv1599367594:84 , srv1607757667:84 , srv1612084952:84 , srv1616726041:84 , srv1631359971:84 , srv1640188300:84 , srv1662940617:84 , srv1679631077:84 , srv1687819568:84 , srv1705108445:84 , srv1772852454:84 , srv1785047078:84 , srv1789335130:84 , srv1790872805:84 , srv1794015980:84 , srv1825317337:84 , srv1830978109:84 , srv186135582:84 , srv1865737340:84 , srv1870552026:84 , srv187135588:84 , srv1880323821:84 , srv1911738799:84 , srv1911988920:84 , srv1913500582:84 , srv1924521044:84 , srv1933151461:84 , srv194979513:84 , srv1974373793:84 , srv2016617282:84 , srv2018499855:84 , srv2066744332:84 , srv2103399566:84 , srv2137604854:84 , srv2142753996:84 , srv2146796590:84 , srv252475614:84 , srv295169903:84 , srv320312357:84 , srv328441853:84 , srv356907265:84 , srv387835491:84 , srv416687327:84 , srv448991485:84 , srv484279297:84 , srv492141994:84 , srv507864928:84 , srv508030808:84 , srv515823721:84 , srv637164423:84 , srv638267923:84 , srv641201831:84 , srv642968890:84 , srv645763914:84 , srv687514139:84 , srv697104771:84 , srv718500783:84 , srv742151663:84 , srv743026037:84 , srv79775170:84 , srv813954601:84 , srv831024091:84 , srv838383203:84 , srv842687178:84 , srv856236797:84 , srv914383534:84 , srv934741362:84 , srv945454414:84 , srv946771141:84 , srv947308261:84 , srv977665906:84 , srv993242529:84 , srv2135580725:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-06-01 15:53:25,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:53:25,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2016617282=56, srv416687327=70, srv637164423=77, srv1933151461=53, srv1092957260=5, srv645763914=81, srv484279297=72, srv515823721=76, srv1913500582=51, srv1830978109=43, srv641201831=79, srv697104771=83, srv387835491=69, srv1825317337=42, srv1176703863=8, srv1911738799=49, srv1180458308=9, srv638267923=78, srv1250327139=14, srv642968890=80, srv1880323821=48, srv1640188300=32, srv2142753996=62, srv252475614=64, srv79775170=87, srv356907265=68, srv1865737340=45, srv2146796590=63, srv1631359971=31, srv492141994=73, srv187135588=47, srv328441853=67, srv813954601=88, srv934741362=94, srv2103399566=59, srv132078957=15, srv1911988920=50, srv106495684=3, srv1790872805=40, srv1342808762=17, srv1363645478=19, srv742151663=85, srv945454414=95, srv1569319342=26, srv1789335130=39, srv743026037=86, srv124368945=13, srv1537024694=25, srv1347998407=18, srv124038085=12, srv15266221=24, srv1870552026=46, srv1374428035=20, srv856236797=92, srv1687819568=35, srv1431219291=22, srv1607757667=28, srv507864928=74, srv1090589006=4, srv1044490786=2, srv842687178=91, srv2018499855=57, srv1612084952=29, srv946771141=96, srv295169903=65, srv1974373793=55, srv977665906=98, srv1225771019=11, srv1924521044=52, srv838383203=90, srv718500783=84, srv831024091=89, srv194979513=54, srv508030808=75, srv1679631077=34, srv1023280128=1, srv186135582=44, srv2137604854=61, srv1512797508=23, srv947308261=97, srv320312357=66, srv993242529=99, srv1599367594=27, srv1662940617=33, srv1772852454=37, srv1102074384=6, srv1212803779=10, srv1330872238=16, srv1705108445=36, srv2066744332=58, srv448991485=71, srv914383534=93, srv1175072163=7, srv1401838630=21, srv1794015980=41, srv1616726041=30, srv2135580725=60, srv1785047078=38, srv1021057827=0, srv687514139=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:53:25,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:53:25,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:53:25,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-01 15:53:25,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-01 15:53:25,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-01 15:53:25,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-01 15:53:25,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-01 15:53:25,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-01 15:53:25,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-01 15:53:25,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-01 15:53:25,936 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:25,937 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-01 15:53:25,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-01 15:53:26,215 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 412 ms to try 5000 different iterations. Found a solution that moves 4708 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1813686435538821. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04070858847599231, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05588789484426352, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5231111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04070858847599231, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17691666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24008333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:26,227 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1023280128:81 , srv1911988920:81 , srv947308261:81 , srv1537024694:82 , srv1785047078:82 , srv2066744332:82 , srv842687178:82 , srv1176703863:83 , srv1212803779:83 , srv1250327139:83 , srv1612084952:83 , srv1790872805:83 , srv1825317337:83 , srv187135588:83 , srv2103399566:83 , srv2142753996:83 , srv416687327:83 , srv637164423:83 , srv742151663:83 , srv1044490786:84 , srv1431219291:84 , srv1569319342:84 , srv1599367594:84 , srv1772852454:84 , srv1870552026:84 , srv320312357:84 , srv492141994:84 , srv645763914:84 , srv831024091:84 , srv106495684:85 , srv1092957260:85 , srv1102074384:85 , srv124038085:85 , srv132078957:85 , srv1347998407:85 , srv1662940617:85 , srv1679631077:85 , srv1789335130:85 , srv1830978109:85 , srv1911738799:85 , srv1974373793:85 , srv2146796590:85 , srv295169903:85 , srv508030808:85 , srv638267923:85 , srv687514139:85 , srv914383534:85 , srv1175072163:86 , srv1342808762:86 , srv1374428035:86 , srv1401838630:86 , srv1512797508:86 , srv1687819568:86 , srv1794015980:86 , srv2016617282:86 , srv328441853:86 , srv697104771:86 , srv79775170:86 , srv945454414:86 , srv977665906:86 , srv124368945:87 , srv1640188300:87 , srv1913500582:87 , srv1933151461:87 , srv194979513:87 , srv2018499855:87 , srv356907265:87 , srv718500783:87 , srv813954601:87 , srv838383203:87 , srv1180458308:88 , srv1225771019:88 , srv1616726041:88 , srv1705108445:88 , srv387835491:88 , srv507864928:88 , srv515823721:88 , srv641201831:88 , srv743026037:88 , srv993242529:88 , srv1363645478:89 , srv15266221:89 , srv186135582:89 , srv1865737340:89 , srv1924521044:89 , srv2137604854:89 , srv252475614:89 , srv946771141:89 , srv1880323821:90 , srv856236797:90 , srv934741362:90 , srv448991485:91 , srv642968890:91 , srv484279297:93 , srv1607757667:94 , srv1021057827:95 , srv1090589006:97 , srv1631359971:99 , srv1330872238:102 , srv2135580725:451 } 2023-06-01 15:53:26,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:53:26,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2016617282=56, srv416687327=70, srv637164423=77, srv1933151461=53, srv1092957260=5, srv645763914=81, srv484279297=72, srv515823721=76, srv1913500582=51, srv1830978109=43, srv641201831=79, srv697104771=83, srv387835491=69, srv1825317337=42, srv1176703863=8, srv1911738799=49, srv1180458308=9, srv638267923=78, srv1250327139=14, srv642968890=80, srv1880323821=48, srv1640188300=32, srv2142753996=62, srv252475614=64, srv79775170=87, srv356907265=68, srv1865737340=45, srv2146796590=63, srv1631359971=31, srv492141994=73, srv187135588=47, srv328441853=67, srv813954601=88, srv934741362=94, srv2103399566=59, srv132078957=15, srv1911988920=50, srv106495684=3, srv1790872805=40, srv1342808762=17, srv1363645478=19, srv742151663=85, srv945454414=95, srv1569319342=26, srv1789335130=39, srv743026037=86, srv124368945=13, srv1537024694=25, srv1347998407=18, srv124038085=12, srv15266221=24, srv1870552026=46, srv1374428035=20, srv856236797=92, srv1687819568=35, srv1431219291=22, srv1607757667=28, srv507864928=74, srv1090589006=4, srv1044490786=2, srv842687178=91, srv2018499855=57, srv1612084952=29, srv946771141=96, srv295169903=65, srv1974373793=55, srv977665906=98, srv1225771019=11, srv1924521044=52, srv838383203=90, srv718500783=84, srv831024091=89, srv194979513=54, srv508030808=75, srv1679631077=34, srv1023280128=1, srv186135582=44, srv2137604854=61, srv1512797508=23, srv947308261=97, srv320312357=66, srv993242529=99, srv1599367594=27, srv1662940617=33, srv1772852454=37, srv1102074384=6, srv1212803779=10, srv1330872238=16, srv1705108445=36, srv2066744332=58, srv448991485=71, srv914383534=93, srv1175072163=7, srv1401838630=21, srv1794015980=41, srv1616726041=30, srv2135580725=60, srv1785047078=38, srv1021057827=0, srv687514139=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:53:26,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:53:26,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:53:26,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:53:26,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:53:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-01 15:53:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-01 15:53:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-01 15:53:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-01 15:53:26,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-01 15:53:26,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-01 15:53:26,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-01 15:53:26,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-01 15:53:26,407 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04070858847599231, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05588789484426352, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04070858847599231, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17691666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24008333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:26,407 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-01 15:53:26,407 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1813356774332831, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04070858847599231, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05588789484426352, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04070858847599231, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17691666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24008333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-01 15:53:26,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 405 ms to try 5000 different iterations. Found a solution that moves 2555 regions; Going from a computed imbalance of 0.1813356774332831 to a new imbalance of 0.015431072343225338. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02215590553652478); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032059848635022606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2838888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02215590553652478); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:26,706 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1330872238:69 , srv1631359971:74 , srv1090589006:76 , srv1607757667:77 , srv1021057827:78 , srv484279297:80 , srv642968890:81 , srv856236797:81 , srv15266221:82 , srv186135582:82 , srv1865737340:82 , srv1924521044:82 , srv252475614:82 , srv448991485:82 , srv934741362:82 , srv1880323821:83 , srv387835491:83 , srv641201831:83 , srv946771141:83 , srv993242529:83 , srv1175072163:84 , srv1225771019:84 , srv1363645478:84 , srv1374428035:84 , srv1401838630:84 , srv1933151461:84 , srv2016617282:84 , srv2018499855:84 , srv356907265:84 , srv697104771:84 , srv743026037:84 , srv813954601:84 , srv1092957260:85 , srv124368945:85 , srv132078957:85 , srv1512797508:85 , srv1616726041:85 , srv1687819568:85 , srv1794015980:85 , srv2137604854:85 , srv515823721:85 , srv79775170:85 , srv838383203:85 , srv1180458308:86 , srv1342808762:86 , srv1640188300:86 , srv1913500582:86 , srv2146796590:86 , srv295169903:86 , srv328441853:86 , srv507864928:86 , srv687514139:86 , srv1044490786:87 , srv106495684:87 , srv1212803779:87 , srv124038085:87 , srv1612084952:87 , srv1662940617:87 , srv1679631077:87 , srv1705108445:87 , srv1911738799:87 , srv194979513:87 , srv2103399566:87 , srv320312357:87 , srv718500783:87 , srv945454414:87 , srv977665906:87 , srv1102074384:88 , srv1347998407:88 , srv1431219291:88 , srv1537024694:88 , srv1569319342:88 , srv1599367594:88 , srv1772852454:88 , srv1790872805:88 , srv1974373793:88 , srv2142753996:88 , srv508030808:88 , srv638267923:88 , srv742151663:88 , srv831024091:88 , srv1176703863:89 , srv1250327139:89 , srv1825317337:89 , srv1830978109:89 , srv187135588:89 , srv637164423:89 , srv842687178:89 , srv914383534:89 , srv1789335130:90 , srv1870552026:90 , srv492141994:90 , srv645763914:90 , srv1023280128:91 , srv1911988920:91 , srv2066744332:91 , srv1785047078:92 , srv416687327:92 , srv947308261:92 , srv2135580725:520 } 2023-06-01 15:53:26,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:53:26,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2016617282=56, srv416687327=70, srv637164423=77, srv1933151461=53, srv1092957260=5, srv645763914=81, srv484279297=72, srv515823721=76, srv1913500582=51, srv1830978109=43, srv641201831=79, srv697104771=83, srv387835491=69, srv1825317337=42, srv1176703863=8, srv1911738799=49, srv1180458308=9, srv638267923=78, srv1250327139=14, srv642968890=80, srv1880323821=48, srv1640188300=32, srv2142753996=62, srv252475614=64, srv79775170=87, srv356907265=68, srv1865737340=45, srv2146796590=63, srv1631359971=31, srv492141994=73, srv187135588=47, srv328441853=67, srv813954601=88, srv934741362=94, srv2103399566=59, srv132078957=15, srv1911988920=50, srv106495684=3, srv1790872805=40, srv1342808762=17, srv1363645478=19, srv742151663=85, srv945454414=95, srv1569319342=26, srv1789335130=39, srv743026037=86, srv124368945=13, srv1537024694=25, srv1347998407=18, srv124038085=12, srv15266221=24, srv1870552026=46, srv1374428035=20, srv856236797=92, srv1687819568=35, srv1431219291=22, srv1607757667=28, srv507864928=74, srv1090589006=4, srv1044490786=2, srv842687178=91, srv2018499855=57, srv1612084952=29, srv946771141=96, srv295169903=65, srv1974373793=55, srv977665906=98, srv1225771019=11, srv1924521044=52, srv838383203=90, srv718500783=84, srv831024091=89, srv194979513=54, srv508030808=75, srv1679631077=34, srv1023280128=1, srv186135582=44, srv2137604854=61, srv1512797508=23, srv947308261=97, srv320312357=66, srv993242529=99, srv1599367594=27, srv1662940617=33, srv1772852454=37, srv1102074384=6, srv1212803779=10, srv1330872238=16, srv1705108445=36, srv2066744332=58, srv448991485=71, srv914383534=93, srv1175072163=7, srv1401838630=21, srv1794015980=41, srv1616726041=30, srv2135580725=60, srv1785047078=38, srv1021057827=0, srv687514139=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-01 15:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-01 15:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-01 15:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-01 15:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-01 15:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-01 15:53:26,899 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02215590553652478); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032059848635022606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02215590553652478); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:26,899 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-01 15:53:26,899 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015413181850844178, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02215590553652478); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032059848635022606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02215590553652478); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-01 15:53:27,260 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 498 ms to try 5000 different iterations. Found a solution that moves 966 regions; Going from a computed imbalance of 0.015413181850844178 to a new imbalance of 9.956438306981527E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006536324013726713); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009356392870229827); MoveCostFunction : (multiplier=7.0, imbalance=0.10733333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006536324013726713); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:27,264 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1705108445:81 , srv1785047078:83 , srv1789335130:83 , srv1830978109:83 , srv416687327:83 , srv492141994:83 , srv507864928:83 , srv645763914:83 , srv718500783:83 , srv1090589006:84 , srv1180458308:84 , srv1631359971:84 , srv1640188300:84 , srv1870552026:84 , srv194979513:84 , srv1974373793:84 , srv2137604854:84 , srv448991485:84 , srv508030808:84 , srv515823721:84 , srv914383534:84 , srv945454414:84 , srv947308261:84 , srv1021057827:85 , srv1023280128:85 , srv106495684:85 , srv1102074384:85 , srv1225771019:85 , srv124038085:85 , srv1342808762:85 , srv1347998407:85 , srv1363645478:85 , srv1431219291:85 , srv1569319342:85 , srv1599367594:85 , srv1662940617:85 , srv1825317337:85 , srv187135588:85 , srv1880323821:85 , srv1911738799:85 , srv1913500582:85 , srv2066744332:85 , srv484279297:85 , srv637164423:85 , srv638267923:85 , srv743026037:85 , srv831024091:85 , srv838383203:85 , srv946771141:85 , srv977665906:85 , srv1044490786:86 , srv1176703863:86 , srv124368945:86 , srv1250327139:86 , srv1330872238:86 , srv1512797508:86 , srv15266221:86 , srv1607757667:86 , srv1616726041:86 , srv1679631077:86 , srv1687819568:86 , srv1772852454:86 , srv186135582:86 , srv1911988920:86 , srv1924521044:86 , srv1933151461:86 , srv2018499855:86 , srv2142753996:86 , srv2146796590:86 , srv252475614:86 , srv295169903:86 , srv320312357:86 , srv328441853:86 , srv387835491:86 , srv642968890:86 , srv813954601:86 , srv856236797:86 , srv934741362:86 , srv1092957260:87 , srv1175072163:87 , srv132078957:87 , srv1374428035:87 , srv1612084952:87 , srv1790872805:87 , srv1794015980:87 , srv1865737340:87 , srv2103399566:87 , srv356907265:87 , srv641201831:87 , srv687514139:87 , srv697104771:87 , srv742151663:87 , srv79775170:87 , srv842687178:87 , srv993242529:87 , srv1212803779:88 , srv1401838630:88 , srv1537024694:88 , srv2016617282:88 , srv2135580725:545 } 2023-06-01 15:53:27,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:53:27,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2016617282=56, srv416687327=70, srv637164423=77, srv1933151461=53, srv1092957260=5, srv645763914=81, srv484279297=72, srv515823721=76, srv1913500582=51, srv1830978109=43, srv641201831=79, srv697104771=83, srv387835491=69, srv1825317337=42, srv1176703863=8, srv1911738799=49, srv1180458308=9, srv638267923=78, srv1250327139=14, srv642968890=80, srv1880323821=48, srv1640188300=32, srv2142753996=62, srv252475614=64, srv79775170=87, srv356907265=68, srv1865737340=45, srv2146796590=63, srv1631359971=31, srv492141994=73, srv187135588=47, srv328441853=67, srv813954601=88, srv934741362=94, srv2103399566=59, srv132078957=15, srv1911988920=50, srv106495684=3, srv1790872805=40, srv1342808762=17, srv1363645478=19, srv742151663=85, srv945454414=95, srv1569319342=26, srv1789335130=39, srv743026037=86, srv124368945=13, srv1537024694=25, srv1347998407=18, srv124038085=12, srv15266221=24, srv1870552026=46, srv1374428035=20, srv856236797=92, srv1687819568=35, srv1431219291=22, srv1607757667=28, srv507864928=74, srv1090589006=4, srv1044490786=2, srv842687178=91, srv2018499855=57, srv1612084952=29, srv946771141=96, srv295169903=65, srv1974373793=55, srv977665906=98, srv1225771019=11, srv1924521044=52, srv838383203=90, srv718500783=84, srv831024091=89, srv194979513=54, srv508030808=75, srv1679631077=34, srv1023280128=1, srv186135582=44, srv2137604854=61, srv1512797508=23, srv947308261=97, srv320312357=66, srv993242529=99, srv1599367594=27, srv1662940617=33, srv1772852454=37, srv1102074384=6, srv1212803779=10, srv1330872238=16, srv1705108445=36, srv2066744332=58, srv448991485=71, srv914383534=93, srv1175072163=7, srv1401838630=21, srv1794015980=41, srv1616726041=30, srv2135580725=60, srv1785047078=38, srv1021057827=0, srv687514139=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:53:27,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-01 15:53:27,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-01 15:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-01 15:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-01 15:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-01 15:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-01 15:53:27,403 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006536324013726713); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009356392870229827); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006536324013726713); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:27,404 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-01 15:53:27,404 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.888797541266453E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006536324013726713); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009356392870229827); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006536324013726713); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-01 15:53:27,719 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 412 ms to try 5000 different iterations. Found a solution that moves 242 regions; Going from a computed imbalance of 9.888797541266453E-4 to a new imbalance of 2.2059677390827604E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.02688888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:27,721 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1616726041:83 , srv842687178:83 , srv1021057827:84 , srv106495684:84 , srv1090589006:84 , srv1102074384:84 , srv1176703863:84 , srv1180458308:84 , srv1212803779:84 , srv124368945:84 , srv1250327139:84 , srv132078957:84 , srv1330872238:84 , srv1342808762:84 , srv1347998407:84 , srv1363645478:84 , srv1374428035:84 , srv1401838630:84 , srv15266221:84 , srv1537024694:84 , srv1679631077:84 , srv1687819568:84 , srv1772852454:84 , srv1790872805:84 , srv1794015980:84 , srv1865737340:84 , srv1870552026:84 , srv1880323821:84 , srv1911988920:84 , srv194979513:84 , srv2016617282:84 , srv2066744332:84 , srv2137604854:84 , srv328441853:84 , srv356907265:84 , srv416687327:84 , srv638267923:84 , srv641201831:84 , srv642968890:84 , srv645763914:84 , srv742151663:84 , srv79775170:84 , srv914383534:84 , srv977665906:84 , srv993242529:84 , srv1023280128:85 , srv1044490786:85 , srv1092957260:85 , srv1175072163:85 , srv124038085:85 , srv1431219291:85 , srv1512797508:85 , srv1569319342:85 , srv1599367594:85 , srv1607757667:85 , srv1631359971:85 , srv1640188300:85 , srv1662940617:85 , srv1785047078:85 , srv1789335130:85 , srv1825317337:85 , srv1830978109:85 , srv186135582:85 , srv187135588:85 , srv1911738799:85 , srv1913500582:85 , srv1924521044:85 , srv1974373793:85 , srv2018499855:85 , srv2103399566:85 , srv2142753996:85 , srv2146796590:85 , srv252475614:85 , srv295169903:85 , srv320312357:85 , srv387835491:85 , srv484279297:85 , srv492141994:85 , srv507864928:85 , srv508030808:85 , srv515823721:85 , srv637164423:85 , srv687514139:85 , srv697104771:85 , srv718500783:85 , srv813954601:85 , srv831024091:85 , srv838383203:85 , srv856236797:85 , srv934741362:85 , srv945454414:85 , srv946771141:85 , srv947308261:85 , srv1225771019:86 , srv1705108445:86 , srv1933151461:86 , srv448991485:86 , srv743026037:86 , srv1612084952:87 , srv2135580725:625 } 2023-06-01 15:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2016617282=56, srv416687327=70, srv637164423=77, srv1933151461=53, srv1092957260=5, srv645763914=81, srv484279297=72, srv515823721=76, srv1913500582=51, srv1830978109=43, srv641201831=79, srv697104771=83, srv387835491=69, srv1825317337=42, srv1176703863=8, srv1911738799=49, srv1180458308=9, srv638267923=78, srv1250327139=14, srv642968890=80, srv1880323821=48, srv1640188300=32, srv2142753996=62, srv252475614=64, srv79775170=87, srv356907265=68, srv1865737340=45, srv2146796590=63, srv1631359971=31, srv492141994=73, srv187135588=47, srv328441853=67, srv813954601=88, srv934741362=94, srv2103399566=59, srv132078957=15, srv1911988920=50, srv106495684=3, srv1790872805=40, srv1342808762=17, srv1363645478=19, srv742151663=85, srv945454414=95, srv1569319342=26, srv1789335130=39, srv743026037=86, srv124368945=13, srv1537024694=25, srv1347998407=18, srv124038085=12, srv15266221=24, srv1870552026=46, srv1374428035=20, srv856236797=92, srv1687819568=35, srv1431219291=22, srv1607757667=28, srv507864928=74, srv1090589006=4, srv1044490786=2, srv842687178=91, srv2018499855=57, srv1612084952=29, srv946771141=96, srv295169903=65, srv1974373793=55, srv977665906=98, srv1225771019=11, srv1924521044=52, srv838383203=90, srv718500783=84, srv831024091=89, srv194979513=54, srv508030808=75, srv1679631077=34, srv1023280128=1, srv186135582=44, srv2137604854=61, srv1512797508=23, srv947308261=97, srv320312357=66, srv993242529=99, srv1599367594=27, srv1662940617=33, srv1772852454=37, srv1102074384=6, srv1212803779=10, srv1330872238=16, srv1705108445=36, srv2066744332=58, srv448991485=71, srv914383534=93, srv1175072163=7, srv1401838630=21, srv1794015980=41, srv1616726041=30, srv2135580725=60, srv1785047078=38, srv1021057827=0, srv687514139=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:53:27,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:53:27,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-01 15:53:27,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-01 15:53:27,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-01 15:53:27,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-01 15:53:27,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-01 15:53:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-01 15:53:27,852 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.036515717312077E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:27,852 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1616726041:83 , srv842687178:83 , srv1021057827:84 , srv106495684:84 , srv1090589006:84 , srv1102074384:84 , srv1176703863:84 , srv1180458308:84 , srv1212803779:84 , srv124368945:84 , srv1250327139:84 , srv132078957:84 , srv1330872238:84 , srv1342808762:84 , srv1347998407:84 , srv1363645478:84 , srv1374428035:84 , srv1401838630:84 , srv15266221:84 , srv1537024694:84 , srv1679631077:84 , srv1687819568:84 , srv1772852454:84 , srv1790872805:84 , srv1794015980:84 , srv1865737340:84 , srv1870552026:84 , srv1880323821:84 , srv1911988920:84 , srv194979513:84 , srv2016617282:84 , srv2066744332:84 , srv2137604854:84 , srv328441853:84 , srv356907265:84 , srv416687327:84 , srv638267923:84 , srv641201831:84 , srv642968890:84 , srv645763914:84 , srv742151663:84 , srv79775170:84 , srv914383534:84 , srv977665906:84 , srv993242529:84 , srv1023280128:85 , srv1044490786:85 , srv1092957260:85 , srv1175072163:85 , srv124038085:85 , srv1431219291:85 , srv1512797508:85 , srv1569319342:85 , srv1599367594:85 , srv1607757667:85 , srv1631359971:85 , srv1640188300:85 , srv1662940617:85 , srv1785047078:85 , srv1789335130:85 , srv1825317337:85 , srv1830978109:85 , srv186135582:85 , srv187135588:85 , srv1911738799:85 , srv1913500582:85 , srv1924521044:85 , srv1974373793:85 , srv2018499855:85 , srv2103399566:85 , srv2142753996:85 , srv2146796590:85 , srv252475614:85 , srv295169903:85 , srv320312357:85 , srv387835491:85 , srv484279297:85 , srv492141994:85 , srv507864928:85 , srv508030808:85 , srv515823721:85 , srv637164423:85 , srv687514139:85 , srv697104771:85 , srv718500783:85 , srv813954601:85 , srv831024091:85 , srv838383203:85 , srv856236797:85 , srv934741362:85 , srv945454414:85 , srv946771141:85 , srv947308261:85 , srv1225771019:86 , srv1705108445:86 , srv1933151461:86 , srv448991485:86 , srv743026037:86 , srv1612084952:87 , srv2135580725:625 } 2023-06-01 15:53:27,883 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 269) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=1714 (was 2308) 2023-06-01 15:53:27,898 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=173, AvailableMemoryMB=1713 2023-06-01 15:53:27,899 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-01 15:53:27,899 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-06-01 15:53:27,912 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1056211112:84 , srv113433854:84 , srv1140656789:84 , srv1284516813:84 , srv1351134615:84 , srv1425496243:84 , srv1494718989:84 , srv155623578:84 , srv1577314615:84 , srv1581891842:84 , srv1673070184:84 , srv1678186870:84 , srv1891035115:84 , srv2018700989:84 , srv2053825202:84 , srv228150998:84 , srv23132883:84 , srv278473177:84 , srv398809283:84 , srv412027064:84 , srv467563705:84 , srv579675762:84 , srv637296752:84 , srv645497693:84 , srv734354376:84 , srv799774846:84 , srv875209436:84 , srv933114874:84 , srv948051174:84 , srv1991549176:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-06-01 15:53:27,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581891842=9, srv23132883=17, srv1284516813=3, srv1991549176=13, srv1140656789=2, srv278473177=18, srv579675762=22, srv113433854=1, srv645497693=24, srv1891035115=12, srv1678186870=11, srv1673070184=10, srv734354376=25, srv933114874=28, srv2053825202=15, srv2018700989=14, srv155623578=7, srv1056211112=0, srv398809283=19, srv875209436=27, srv948051174=29, srv1425496243=5, srv412027064=20, srv1494718989=6, srv467563705=21, srv799774846=26, srv637296752=23, srv1577314615=8, srv1351134615=4, srv228150998=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:53:27,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:53:27,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:53:27,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-01 15:53:27,955 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.126077799259273, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:53:27,956 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912590310292927, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.126077799259273, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-06-01 15:55:27,934 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5783326 different iterations. Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912590310292927 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:55:27,940 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1056211112:90 , srv113433854:90 , srv1140656789:90 , srv1284516813:90 , srv1351134615:90 , srv1425496243:90 , srv1494718989:90 , srv155623578:90 , srv1577314615:90 , srv1581891842:90 , srv1673070184:90 , srv1678186870:90 , srv1891035115:90 , srv1991549176:90 , srv2018700989:90 , srv2053825202:90 , srv228150998:90 , srv23132883:90 , srv278473177:90 , srv398809283:90 , srv412027064:90 , srv467563705:90 , srv579675762:90 , srv637296752:90 , srv645497693:90 , srv734354376:90 , srv799774846:90 , srv875209436:90 , srv933114874:90 , srv948051174:90 } 2023-06-01 15:55:27,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-01 15:55:27,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581891842=9, srv23132883=17, srv1284516813=3, srv1991549176=13, srv1140656789=2, srv278473177=18, srv579675762=22, srv113433854=1, srv645497693=24, srv1891035115=12, srv1678186870=11, srv1673070184=10, srv734354376=25, srv933114874=28, srv2053825202=15, srv2018700989=14, srv155623578=7, srv1056211112=0, srv398809283=19, srv875209436=27, srv948051174=29, srv1425496243=5, srv412027064=20, srv1494718989=6, srv467563705=21, srv799774846=26, srv637296752=23, srv1577314615=8, srv1351134615=4, srv228150998=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-01 15:55:27,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-01 15:55:27,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-01 15:55:27,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-01 15:55:27,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-01 15:55:27,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-01 15:55:27,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-01 15:55:27,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-01 15:55:27,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-01 15:55:27,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-01 15:55:27,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-01 15:55:27,957 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-01 15:55:27,958 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1056211112:90 , srv113433854:90 , srv1140656789:90 , srv1284516813:90 , srv1351134615:90 , srv1425496243:90 , srv1494718989:90 , srv155623578:90 , srv1577314615:90 , srv1581891842:90 , srv1673070184:90 , srv1678186870:90 , srv1891035115:90 , srv1991549176:90 , srv2018700989:90 , srv2053825202:90 , srv228150998:90 , srv23132883:90 , srv278473177:90 , srv398809283:90 , srv412027064:90 , srv467563705:90 , srv579675762:90 , srv637296752:90 , srv645497693:90 , srv734354376:90 , srv799774846:90 , srv875209436:90 , srv933114874:90 , srv948051174:90 } 2023-06-01 15:55:27,971 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=243 (was 296), ProcessCount=173 (was 173), AvailableMemoryMB=1126 (was 1713)