2023-04-27 16:53:28,232 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-27 16:53:28,558 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 16:53:28,660 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-27 16:53:28,718 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-27 16:53:28,724 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-27 16:53:28,735 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=404, ProcessCount=174, AvailableMemoryMB=13131 2023-04-27 16:53:28,737 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-27 16:53:28,737 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-27 16:53:28,876 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001437715:84 , srv100896929:84 , srv1099543650:84 , srv1105083569:84 , srv1110164739:84 , srv111740917:84 , srv1135247558:84 , srv1149653745:84 , srv1160491434:84 , srv1171794334:84 , srv1205466501:84 , srv1224786456:84 , srv1231633506:84 , srv123588813:84 , srv1270634363:84 , srv1271928355:84 , srv1293781570:84 , srv1300354248:84 , srv131878857:84 , srv1326137875:84 , srv1380673106:84 , srv1382313885:84 , srv1396272945:84 , srv1398863533:84 , srv1425566770:84 , srv1426839455:84 , srv1434529438:84 , srv1482191929:84 , srv1489234593:84 , srv1491446847:84 , srv1492673743:84 , srv149643422:84 , srv1507111520:84 , srv1527611250:84 , srv1541000897:84 , srv1554784836:84 , srv1585986058:84 , srv1586450445:84 , srv1612346873:84 , srv1616947090:84 , srv1621553740:84 , srv1642295452:84 , srv1705356376:84 , srv1734579261:84 , srv1752473327:84 , srv1809140108:84 , srv1817780602:84 , srv1817789783:84 , srv182774938:84 , srv1852695042:84 , srv1855839203:84 , srv1887148608:84 , srv18886883:84 , srv1892339474:84 , srv1899607014:84 , srv1906893473:84 , srv1943421026:84 , srv200824368:84 , srv2030552725:84 , srv2037374942:84 , srv2048860747:84 , srv2065573298:84 , srv2084360574:84 , srv250994874:84 , srv33272950:84 , srv350523867:84 , srv3556154:84 , srv358225339:84 , srv401033541:84 , srv430750855:84 , srv462615631:84 , srv469293564:84 , srv533151363:84 , srv535138397:84 , srv540807399:84 , srv551664155:84 , srv556679312:84 , srv562498590:84 , srv620494967:84 , srv672884055:84 , srv697294916:84 , srv699003911:84 , srv730991354:84 , srv734542403:84 , srv749195079:84 , srv757896292:84 , srv791638851:84 , srv80312706:84 , srv81128423:84 , srv829239670:84 , srv858256983:84 , srv878400:84 , srv88038396:84 , srv882123313:84 , srv897288933:84 , srv921067020:84 , srv92126865:84 , srv969824632:84 , srv994269566:84 , srv1182615285:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-27 16:53:28,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:53:28,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv533151363=73, srv1482191929=28, srv1554784836=36, srv2037374942=60, srv430750855=70, srv358225339=68, srv1817780602=47, srv1507111520=33, srv994269566=99, srv111740917=5, srv33272950=65, srv2065573298=62, srv1326137875=20, srv462615631=71, srv100896929=1, srv1906893473=56, srv1135247558=6, srv18886883=53, srv131878857=19, srv1541000897=35, srv1892339474=54, srv921067020=96, srv200824368=58, srv401033541=69, srv1899607014=55, srv1398863533=24, srv1270634363=15, srv149643422=32, srv1382313885=22, srv350523867=66, srv1426839455=26, srv540807399=75, srv1612346873=39, srv2084360574=63, srv1205466501=11, srv1492673743=31, srv80312706=88, srv1734579261=44, srv1943421026=57, srv791638851=87, srv1809140108=46, srv1527611250=34, srv829239670=90, srv1160491434=8, srv1705356376=43, srv1586450445=38, srv1752473327=45, srv1110164739=4, srv1271928355=16, srv1855839203=51, srv92126865=97, srv1621553740=41, srv1293781570=17, srv2048860747=61, srv1491446847=30, srv757896292=86, srv858256983=91, srv734542403=84, srv882123313=94, srv1434529438=27, srv1231633506=13, srv1171794334=9, srv1489234593=29, srv730991354=83, srv1425566770=25, srv1001437715=0, srv562498590=78, srv1380673106=21, srv469293564=72, srv1396272945=23, srv897288933=95, srv1852695042=50, srv699003911=82, srv1300354248=18, srv1642295452=42, srv88038396=93, srv1099543650=2, srv3556154=67, srv749195079=85, srv551664155=76, srv1616947090=40, srv81128423=89, srv1105083569=3, srv1224786456=12, srv535138397=74, srv250994874=64, srv1182615285=10, srv182774938=49, srv672884055=80, srv556679312=77, srv878400=92, srv620494967=79, srv969824632=98, srv1585986058=37, srv123588813=14, srv1149653745=7, srv1887148608=52, srv2030552725=59, srv697294916=81, srv1817789783=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:53:28,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:53:28,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:53:28,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-27 16:53:28,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-27 16:53:28,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-27 16:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-27 16:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-27 16:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-27 16:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-27 16:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-27 16:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-27 16:53:29,320 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:29,321 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-27 16:53:29,321 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-27 16:53:29,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 641 ms to try 5000 different iterations. Found a solution that moves 4686 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18173570540322126. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052652590166792, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054327848765758725, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5206666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052652590166792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17683333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24508333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:29,679 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv462615631:81 , srv1855839203:82 , srv2030552725:82 , srv1160491434:83 , srv1434529438:83 , srv18886883:83 , srv1943421026:83 , srv250994874:83 , srv3556154:83 , srv358225339:83 , srv430750855:83 , srv969824632:83 , srv1001437715:84 , srv100896929:84 , srv1135247558:84 , srv1382313885:84 , srv1398863533:84 , srv1809140108:84 , srv1817789783:84 , srv556679312:84 , srv562498590:84 , srv697294916:84 , srv858256983:84 , srv88038396:84 , srv994269566:84 , srv1105083569:85 , srv1205466501:85 , srv131878857:85 , srv1491446847:85 , srv1642295452:85 , srv1734579261:85 , srv1817780602:85 , srv1887148608:85 , srv1906893473:85 , srv350523867:85 , srv401033541:85 , srv533151363:85 , srv540807399:85 , srv672884055:85 , srv80312706:85 , srv882123313:85 , srv1099543650:86 , srv1171794334:86 , srv1224786456:86 , srv1293781570:86 , srv1326137875:86 , srv1489234593:86 , srv1492673743:86 , srv1541000897:86 , srv1586450445:86 , srv1612346873:86 , srv182774938:86 , srv1899607014:86 , srv2048860747:86 , srv535138397:86 , srv551664155:86 , srv620494967:86 , srv699003911:86 , srv730991354:86 , srv734542403:86 , srv791638851:86 , srv81128423:86 , srv921067020:86 , srv92126865:86 , srv1110164739:87 , srv1149653745:87 , srv1270634363:87 , srv1396272945:87 , srv1527611250:87 , srv1705356376:87 , srv1892339474:87 , srv2037374942:87 , srv33272950:87 , srv757896292:87 , srv878400:87 , srv897288933:87 , srv111740917:88 , srv123588813:88 , srv1271928355:88 , srv1300354248:88 , srv1852695042:88 , srv2084360574:88 , srv1507111520:89 , srv1585986058:89 , srv1554784836:90 , srv1616947090:90 , srv749195079:90 , srv1380673106:91 , srv149643422:91 , srv1621553740:91 , srv1752473327:91 , srv2065573298:91 , srv469293564:92 , srv1231633506:93 , srv1482191929:93 , srv200824368:93 , srv1425566770:94 , srv1426839455:94 , srv829239670:94 , srv1182615285:450 } 2023-04-27 16:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv533151363=73, srv1482191929=28, srv1554784836=36, srv2037374942=60, srv430750855=70, srv358225339=68, srv1817780602=47, srv1507111520=33, srv994269566=99, srv111740917=5, srv33272950=65, srv2065573298=62, srv1326137875=20, srv462615631=71, srv100896929=1, srv1906893473=56, srv1135247558=6, srv18886883=53, srv131878857=19, srv1541000897=35, srv1892339474=54, srv921067020=96, srv200824368=58, srv401033541=69, srv1899607014=55, srv1398863533=24, srv1270634363=15, srv149643422=32, srv1382313885=22, srv350523867=66, srv1426839455=26, srv540807399=75, srv1612346873=39, srv2084360574=63, srv1205466501=11, srv1492673743=31, srv80312706=88, srv1734579261=44, srv1943421026=57, srv791638851=87, srv1809140108=46, srv1527611250=34, srv829239670=90, srv1160491434=8, srv1705356376=43, srv1586450445=38, srv1752473327=45, srv1110164739=4, srv1271928355=16, srv1855839203=51, srv92126865=97, srv1621553740=41, srv1293781570=17, srv2048860747=61, srv1491446847=30, srv757896292=86, srv858256983=91, srv734542403=84, srv882123313=94, srv1434529438=27, srv1231633506=13, srv1171794334=9, srv1489234593=29, srv730991354=83, srv1425566770=25, srv1001437715=0, srv562498590=78, srv1380673106=21, srv469293564=72, srv1396272945=23, srv897288933=95, srv1852695042=50, srv699003911=82, srv1300354248=18, srv1642295452=42, srv88038396=93, srv1099543650=2, srv3556154=67, srv749195079=85, srv551664155=76, srv1616947090=40, srv81128423=89, srv1105083569=3, srv1224786456=12, srv535138397=74, srv250994874=64, srv1182615285=10, srv182774938=49, srv672884055=80, srv556679312=77, srv878400=92, srv620494967=79, srv969824632=98, srv1585986058=37, srv123588813=14, srv1149653745=7, srv1887148608=52, srv2030552725=59, srv697294916=81, srv1817789783=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-27 16:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-27 16:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-27 16:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-27 16:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-27 16:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-27 16:53:29,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-27 16:53:29,827 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052652590166792, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054327848765758725, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052652590166792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17683333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24508333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:29,827 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-27 16:53:29,827 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18170289332991474, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052652590166792, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054327848765758725, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052652590166792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17683333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24508333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-27 16:53:30,016 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 293 ms to try 5000 different iterations. Found a solution that moves 2617 regions; Going from a computed imbalance of 0.18170289332991474 to a new imbalance of 0.015563621190385735. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021839007810050513); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035082074683309386, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2907777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021839007810050513); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,024 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1426839455:76 , srv1425566770:79 , srv829239670:79 , srv149643422:80 , srv200824368:80 , srv1231633506:81 , srv1380673106:81 , srv1482191929:81 , srv1554784836:81 , srv1752473327:81 , srv469293564:81 , srv2065573298:82 , srv1149653745:83 , srv123588813:83 , srv1507111520:83 , srv1621553740:83 , srv1899607014:83 , srv2084360574:83 , srv749195079:83 , srv111740917:84 , srv1171794334:84 , srv1300354248:84 , srv1396272945:84 , srv1489234593:84 , srv1527611250:84 , srv1616947090:84 , srv1852695042:84 , srv535138397:84 , srv551664155:84 , srv697294916:84 , srv757896292:84 , srv1099543650:85 , srv1110164739:85 , srv1224786456:85 , srv1270634363:85 , srv1293781570:85 , srv1585986058:85 , srv1612346873:85 , srv1705356376:85 , srv1892339474:85 , srv2037374942:85 , srv80312706:85 , srv878400:85 , srv897288933:85 , srv921067020:85 , srv1001437715:86 , srv1135247558:86 , srv1271928355:86 , srv1326137875:86 , srv1491446847:86 , srv1492673743:86 , srv1541000897:86 , srv1586450445:86 , srv3556154:86 , srv734542403:86 , srv791638851:86 , srv81128423:86 , srv1382313885:87 , srv1642295452:87 , srv1809140108:87 , srv1817780602:87 , srv182774938:87 , srv1887148608:87 , srv350523867:87 , srv430750855:87 , srv672884055:87 , srv699003911:87 , srv730991354:87 , srv1105083569:88 , srv1160491434:88 , srv1398863533:88 , srv1434529438:88 , srv1734579261:88 , srv1817789783:88 , srv18886883:88 , srv1943421026:88 , srv2048860747:88 , srv33272950:88 , srv540807399:88 , srv556679312:88 , srv562498590:88 , srv620494967:88 , srv92126865:88 , srv969824632:88 , srv994269566:88 , srv100896929:89 , srv1205466501:89 , srv1855839203:89 , srv1906893473:89 , srv2030552725:89 , srv358225339:89 , srv88038396:89 , srv882123313:89 , srv250994874:90 , srv462615631:90 , srv858256983:90 , srv401033541:91 , srv533151363:91 , srv131878857:92 , srv1182615285:518 } 2023-04-27 16:53:30,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:53:30,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv533151363=73, srv1482191929=28, srv1554784836=36, srv2037374942=60, srv430750855=70, srv358225339=68, srv1817780602=47, srv1507111520=33, srv994269566=99, srv111740917=5, srv33272950=65, srv2065573298=62, srv1326137875=20, srv462615631=71, srv100896929=1, srv1906893473=56, srv1135247558=6, srv18886883=53, srv131878857=19, srv1541000897=35, srv1892339474=54, srv921067020=96, srv200824368=58, srv401033541=69, srv1899607014=55, srv1398863533=24, srv1270634363=15, srv149643422=32, srv1382313885=22, srv350523867=66, srv1426839455=26, srv540807399=75, srv1612346873=39, srv2084360574=63, srv1205466501=11, srv1492673743=31, srv80312706=88, srv1734579261=44, srv1943421026=57, srv791638851=87, srv1809140108=46, srv1527611250=34, srv829239670=90, srv1160491434=8, srv1705356376=43, srv1586450445=38, srv1752473327=45, srv1110164739=4, srv1271928355=16, srv1855839203=51, srv92126865=97, srv1621553740=41, srv1293781570=17, srv2048860747=61, srv1491446847=30, srv757896292=86, srv858256983=91, srv734542403=84, srv882123313=94, srv1434529438=27, srv1231633506=13, srv1171794334=9, srv1489234593=29, srv730991354=83, srv1425566770=25, srv1001437715=0, srv562498590=78, srv1380673106=21, srv469293564=72, srv1396272945=23, srv897288933=95, srv1852695042=50, srv699003911=82, srv1300354248=18, srv1642295452=42, srv88038396=93, srv1099543650=2, srv3556154=67, srv749195079=85, srv551664155=76, srv1616947090=40, srv81128423=89, srv1105083569=3, srv1224786456=12, srv535138397=74, srv250994874=64, srv1182615285=10, srv182774938=49, srv672884055=80, srv556679312=77, srv878400=92, srv620494967=79, srv969824632=98, srv1585986058=37, srv123588813=14, srv1149653745=7, srv1887148608=52, srv2030552725=59, srv697294916=81, srv1817789783=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:53:30,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:53:30,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-27 16:53:30,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-27 16:53:30,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-27 16:53:30,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-27 16:53:30,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-27 16:53:30,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-27 16:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-27 16:53:30,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021839007810050513); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035082074683309386, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021839007810050513); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,136 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-27 16:53:30,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015545296564725657, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021839007810050513); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035082074683309386, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021839007810050513); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-27 16:53:30,312 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 249 ms to try 5000 different iterations. Found a solution that moves 1016 regions; Going from a computed imbalance of 0.015545296564725657 to a new imbalance of 5.194259785778362E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823037785225997); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010005610098618013); MoveCostFunction : (multiplier=7.0, imbalance=0.11288888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00823037785225997); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,314 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv131878857:80 , srv533151363:80 , srv401033541:81 , srv882123313:82 , srv1231633506:83 , srv1271928355:83 , srv1482191929:83 , srv1585986058:83 , srv1621553740:83 , srv1906893473:83 , srv2048860747:83 , srv33272950:83 , srv620494967:83 , srv829239670:83 , srv100896929:84 , srv1205466501:84 , srv1734579261:84 , srv1817789783:84 , srv182774938:84 , srv200824368:84 , srv2037374942:84 , srv2065573298:84 , srv350523867:84 , srv540807399:84 , srv699003911:84 , srv730991354:84 , srv858256983:84 , srv111740917:85 , srv1270634363:85 , srv1300354248:85 , srv1380673106:85 , srv1398863533:85 , srv1425566770:85 , srv1586450445:85 , srv1612346873:85 , srv1616947090:85 , srv1705356376:85 , srv1752473327:85 , srv1852695042:85 , srv1887148608:85 , srv1892339474:85 , srv250994874:85 , srv358225339:85 , srv469293564:85 , srv556679312:85 , srv562498590:85 , srv672884055:85 , srv734542403:85 , srv749195079:85 , srv791638851:85 , srv88038396:85 , srv92126865:85 , srv969824632:85 , srv994269566:85 , srv1099543650:86 , srv1105083569:86 , srv1160491434:86 , srv1224786456:86 , srv123588813:86 , srv1293781570:86 , srv1326137875:86 , srv1382313885:86 , srv1396272945:86 , srv1434529438:86 , srv1491446847:86 , srv1507111520:86 , srv1527611250:86 , srv1541000897:86 , srv1642295452:86 , srv1809140108:86 , srv1817780602:86 , srv1855839203:86 , srv2030552725:86 , srv2084360574:86 , srv462615631:86 , srv551664155:86 , srv81128423:86 , srv878400:86 , srv897288933:86 , srv921067020:86 , srv1110164739:87 , srv1149653745:87 , srv1171794334:87 , srv1426839455:87 , srv1489234593:87 , srv1492673743:87 , srv149643422:87 , srv1554784836:87 , srv18886883:87 , srv1943421026:87 , srv430750855:87 , srv535138397:87 , srv757896292:87 , srv80312706:87 , srv1135247558:88 , srv3556154:88 , srv1001437715:89 , srv1899607014:89 , srv697294916:89 , srv1182615285:563 } 2023-04-27 16:53:30,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:53:30,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv533151363=73, srv1482191929=28, srv1554784836=36, srv2037374942=60, srv430750855=70, srv358225339=68, srv1817780602=47, srv1507111520=33, srv994269566=99, srv111740917=5, srv33272950=65, srv2065573298=62, srv1326137875=20, srv462615631=71, srv100896929=1, srv1906893473=56, srv1135247558=6, srv18886883=53, srv131878857=19, srv1541000897=35, srv1892339474=54, srv921067020=96, srv200824368=58, srv401033541=69, srv1899607014=55, srv1398863533=24, srv1270634363=15, srv149643422=32, srv1382313885=22, srv350523867=66, srv1426839455=26, srv540807399=75, srv1612346873=39, srv2084360574=63, srv1205466501=11, srv1492673743=31, srv80312706=88, srv1734579261=44, srv1943421026=57, srv791638851=87, srv1809140108=46, srv1527611250=34, srv829239670=90, srv1160491434=8, srv1705356376=43, srv1586450445=38, srv1752473327=45, srv1110164739=4, srv1271928355=16, srv1855839203=51, srv92126865=97, srv1621553740=41, srv1293781570=17, srv2048860747=61, srv1491446847=30, srv757896292=86, srv858256983=91, srv734542403=84, srv882123313=94, srv1434529438=27, srv1231633506=13, srv1171794334=9, srv1489234593=29, srv730991354=83, srv1425566770=25, srv1001437715=0, srv562498590=78, srv1380673106=21, srv469293564=72, srv1396272945=23, srv897288933=95, srv1852695042=50, srv699003911=82, srv1300354248=18, srv1642295452=42, srv88038396=93, srv1099543650=2, srv3556154=67, srv749195079=85, srv551664155=76, srv1616947090=40, srv81128423=89, srv1105083569=3, srv1224786456=12, srv535138397=74, srv250994874=64, srv1182615285=10, srv182774938=49, srv672884055=80, srv556679312=77, srv878400=92, srv620494967=79, srv969824632=98, srv1585986058=37, srv123588813=14, srv1149653745=7, srv1887148608=52, srv2030552725=59, srv697294916=81, srv1817789783=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-27 16:53:30,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-27 16:53:30,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-27 16:53:30,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-27 16:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-27 16:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-27 16:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-27 16:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-27 16:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-27 16:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-27 16:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-27 16:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-27 16:53:30,431 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823037785225997); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010005610098618013); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00823037785225997); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,431 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-27 16:53:30,431 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.123117945233316E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823037785225997); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010005610098618013); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00823037785225997); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-27 16:53:30,673 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 323 ms to try 5000 different iterations. Found a solution that moves 253 regions; Going from a computed imbalance of 5.123117945233316E-4 to a new imbalance of 2.136493711733189E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.02811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,674 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1001437715:83 , srv1105083569:83 , srv1110164739:83 , srv1616947090:83 , srv92126865:83 , srv1135247558:84 , srv1205466501:84 , srv1300354248:84 , srv1326137875:84 , srv1398863533:84 , srv1492673743:84 , srv149643422:84 , srv1507111520:84 , srv1541000897:84 , srv1554784836:84 , srv1642295452:84 , srv1817780602:84 , srv1943421026:84 , srv250994874:84 , srv33272950:84 , srv469293564:84 , srv699003911:84 , srv749195079:84 , srv757896292:84 , srv81128423:84 , srv858256983:84 , srv878400:84 , srv88038396:84 , srv897288933:84 , srv100896929:85 , srv1099543650:85 , srv111740917:85 , srv1149653745:85 , srv1160491434:85 , srv1171794334:85 , srv1224786456:85 , srv1231633506:85 , srv123588813:85 , srv1270634363:85 , srv1271928355:85 , srv1293781570:85 , srv131878857:85 , srv1380673106:85 , srv1382313885:85 , srv1396272945:85 , srv1425566770:85 , srv1426839455:85 , srv1434529438:85 , srv1482191929:85 , srv1489234593:85 , srv1491446847:85 , srv1527611250:85 , srv1585986058:85 , srv1586450445:85 , srv1621553740:85 , srv1705356376:85 , srv1734579261:85 , srv1809140108:85 , srv182774938:85 , srv1852695042:85 , srv1855839203:85 , srv1887148608:85 , srv18886883:85 , srv1892339474:85 , srv1899607014:85 , srv1906893473:85 , srv200824368:85 , srv2030552725:85 , srv2048860747:85 , srv2065573298:85 , srv2084360574:85 , srv3556154:85 , srv358225339:85 , srv430750855:85 , srv462615631:85 , srv535138397:85 , srv540807399:85 , srv551664155:85 , srv556679312:85 , srv562498590:85 , srv620494967:85 , srv672884055:85 , srv697294916:85 , srv730991354:85 , srv734542403:85 , srv791638851:85 , srv80312706:85 , srv882123313:85 , srv921067020:85 , srv969824632:85 , srv994269566:85 , srv1612346873:86 , srv1752473327:86 , srv1817789783:86 , srv2037374942:86 , srv350523867:86 , srv401033541:86 , srv533151363:86 , srv829239670:86 , srv1182615285:611 } 2023-04-27 16:53:30,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv533151363=73, srv1482191929=28, srv1554784836=36, srv2037374942=60, srv430750855=70, srv358225339=68, srv1817780602=47, srv1507111520=33, srv994269566=99, srv111740917=5, srv33272950=65, srv2065573298=62, srv1326137875=20, srv462615631=71, srv100896929=1, srv1906893473=56, srv1135247558=6, srv18886883=53, srv131878857=19, srv1541000897=35, srv1892339474=54, srv921067020=96, srv200824368=58, srv401033541=69, srv1899607014=55, srv1398863533=24, srv1270634363=15, srv149643422=32, srv1382313885=22, srv350523867=66, srv1426839455=26, srv540807399=75, srv1612346873=39, srv2084360574=63, srv1205466501=11, srv1492673743=31, srv80312706=88, srv1734579261=44, srv1943421026=57, srv791638851=87, srv1809140108=46, srv1527611250=34, srv829239670=90, srv1160491434=8, srv1705356376=43, srv1586450445=38, srv1752473327=45, srv1110164739=4, srv1271928355=16, srv1855839203=51, srv92126865=97, srv1621553740=41, srv1293781570=17, srv2048860747=61, srv1491446847=30, srv757896292=86, srv858256983=91, srv734542403=84, srv882123313=94, srv1434529438=27, srv1231633506=13, srv1171794334=9, srv1489234593=29, srv730991354=83, srv1425566770=25, srv1001437715=0, srv562498590=78, srv1380673106=21, srv469293564=72, srv1396272945=23, srv897288933=95, srv1852695042=50, srv699003911=82, srv1300354248=18, srv1642295452=42, srv88038396=93, srv1099543650=2, srv3556154=67, srv749195079=85, srv551664155=76, srv1616947090=40, srv81128423=89, srv1105083569=3, srv1224786456=12, srv535138397=74, srv250994874=64, srv1182615285=10, srv182774938=49, srv672884055=80, srv556679312=77, srv878400=92, srv620494967=79, srv969824632=98, srv1585986058=37, srv123588813=14, srv1149653745=7, srv1887148608=52, srv2030552725=59, srv697294916=81, srv1817789783=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:53:30,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:53:30,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:53:30,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:53:30,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:53:30,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:53:30,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-27 16:53:30,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-27 16:53:30,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-27 16:53:30,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-27 16:53:30,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-27 16:53:30,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-27 16:53:30,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-27 16:53:30,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-27 16:53:30,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-27 16:53:30,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-27 16:53:30,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-27 16:53:30,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-27 16:53:30,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-27 16:53:30,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-27 16:53:30,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-27 16:53:30,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.959339325336565E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,800 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1001437715:83 , srv1105083569:83 , srv1110164739:83 , srv1616947090:83 , srv92126865:83 , srv1135247558:84 , srv1205466501:84 , srv1300354248:84 , srv1326137875:84 , srv1398863533:84 , srv1492673743:84 , srv149643422:84 , srv1507111520:84 , srv1541000897:84 , srv1554784836:84 , srv1642295452:84 , srv1817780602:84 , srv1943421026:84 , srv250994874:84 , srv33272950:84 , srv469293564:84 , srv699003911:84 , srv749195079:84 , srv757896292:84 , srv81128423:84 , srv858256983:84 , srv878400:84 , srv88038396:84 , srv897288933:84 , srv100896929:85 , srv1099543650:85 , srv111740917:85 , srv1149653745:85 , srv1160491434:85 , srv1171794334:85 , srv1224786456:85 , srv1231633506:85 , srv123588813:85 , srv1270634363:85 , srv1271928355:85 , srv1293781570:85 , srv131878857:85 , srv1380673106:85 , srv1382313885:85 , srv1396272945:85 , srv1425566770:85 , srv1426839455:85 , srv1434529438:85 , srv1482191929:85 , srv1489234593:85 , srv1491446847:85 , srv1527611250:85 , srv1585986058:85 , srv1586450445:85 , srv1621553740:85 , srv1705356376:85 , srv1734579261:85 , srv1809140108:85 , srv182774938:85 , srv1852695042:85 , srv1855839203:85 , srv1887148608:85 , srv18886883:85 , srv1892339474:85 , srv1899607014:85 , srv1906893473:85 , srv200824368:85 , srv2030552725:85 , srv2048860747:85 , srv2065573298:85 , srv2084360574:85 , srv3556154:85 , srv358225339:85 , srv430750855:85 , srv462615631:85 , srv535138397:85 , srv540807399:85 , srv551664155:85 , srv556679312:85 , srv562498590:85 , srv620494967:85 , srv672884055:85 , srv697294916:85 , srv730991354:85 , srv734542403:85 , srv791638851:85 , srv80312706:85 , srv882123313:85 , srv921067020:85 , srv969824632:85 , srv994269566:85 , srv1612346873:86 , srv1752473327:86 , srv1817789783:86 , srv2037374942:86 , srv350523867:86 , srv401033541:86 , srv533151363:86 , srv829239670:86 , srv1182615285:611 } 2023-04-27 16:53:30,820 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=404 (was 404), ProcessCount=174 (was 174), AvailableMemoryMB=12482 (was 13131) 2023-04-27 16:53:30,828 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=404, ProcessCount=174, AvailableMemoryMB=12482 2023-04-27 16:53:30,828 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-27 16:53:30,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-27 16:53:30,840 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1022856493:84 , srv1091343215:84 , srv1127261237:84 , srv121447842:84 , srv1249501521:84 , srv1356442393:84 , srv1381543823:84 , srv1470238446:84 , srv1539892663:84 , srv1541034409:84 , srv1609342903:84 , srv1621586418:84 , srv1735149095:84 , srv1791160315:84 , srv1855911147:84 , srv1896731675:84 , srv1970053866:84 , srv1992534448:84 , srv330594808:84 , srv46090494:84 , srv477405909:84 , srv561496123:84 , srv610147879:84 , srv624902047:84 , srv713452194:84 , srv794180605:84 , srv831062966:84 , srv948149165:84 , srv986637230:84 , srv809151146:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-27 16:53:30,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:53:30,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1791160315=13, srv1992534448=17, srv1855911147=14, srv1896731675=15, srv1970053866=16, srv1539892663=8, srv1609342903=10, srv986637230=29, srv46090494=19, srv948149165=28, srv794180605=25, srv477405909=20, srv831062966=27, srv624902047=23, srv1381543823=6, srv1541034409=9, srv1022856493=0, srv1249501521=4, srv809151146=26, srv610147879=22, srv1356442393=5, srv1091343215=1, srv1621586418=11, srv561496123=21, srv1735149095=12, srv1470238446=7, srv121447842=3, srv713452194=24, srv1127261237=2, srv330594808=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:53:30,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:53:30,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:53:30,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-27 16:53:30,865 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1121307692203584, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:53:30,865 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912546363656687, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1121307692203584, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-27 16:55:30,855 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9678737 different iterations. Found a solution that moves 1834 regions; Going from a computed imbalance of 0.9912546363656687 to a new imbalance of 4.280647492113412E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6792592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:55:30,862 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1022856493:90 , srv1091343215:90 , srv1127261237:90 , srv121447842:90 , srv1249501521:90 , srv1356442393:90 , srv1381543823:90 , srv1470238446:90 , srv1539892663:90 , srv1541034409:90 , srv1609342903:90 , srv1621586418:90 , srv1735149095:90 , srv1791160315:90 , srv1855911147:90 , srv1896731675:90 , srv1970053866:90 , srv1992534448:90 , srv330594808:90 , srv46090494:90 , srv477405909:90 , srv561496123:90 , srv610147879:90 , srv624902047:90 , srv713452194:90 , srv794180605:90 , srv809151146:90 , srv831062966:90 , srv948149165:90 , srv986637230:90 } 2023-04-27 16:55:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-27 16:55:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1791160315=13, srv1992534448=17, srv1855911147=14, srv1896731675=15, srv1970053866=16, srv1539892663=8, srv1609342903=10, srv986637230=29, srv46090494=19, srv948149165=28, srv794180605=25, srv477405909=20, srv831062966=27, srv624902047=23, srv1381543823=6, srv1541034409=9, srv1022856493=0, srv1249501521=4, srv809151146=26, srv610147879=22, srv1356442393=5, srv1091343215=1, srv1621586418=11, srv561496123=21, srv1735149095=12, srv1470238446=7, srv121447842=3, srv713452194=24, srv1127261237=2, srv330594808=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-27 16:55:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-27 16:55:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-27 16:55:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-27 16:55:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-27 16:55:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-27 16:55:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-27 16:55:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-27 16:55:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-27 16:55:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-27 16:55:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-27 16:55:30,876 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-27 16:55:30,876 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1022856493:90 , srv1091343215:90 , srv1127261237:90 , srv121447842:90 , srv1249501521:90 , srv1356442393:90 , srv1381543823:90 , srv1470238446:90 , srv1539892663:90 , srv1541034409:90 , srv1609342903:90 , srv1621586418:90 , srv1735149095:90 , srv1791160315:90 , srv1855911147:90 , srv1896731675:90 , srv1970053866:90 , srv1992534448:90 , srv330594808:90 , srv46090494:90 , srv477405909:90 , srv561496123:90 , srv610147879:90 , srv624902047:90 , srv713452194:90 , srv794180605:90 , srv809151146:90 , srv831062966:90 , srv948149165:90 , srv986637230:90 } 2023-04-27 16:55:30,886 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 404), ProcessCount=171 (was 174), AvailableMemoryMB=11585 (was 12482)