2023-03-13 05:54:14,957 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-13 05:54:15,307 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-13 05:54:15,413 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-13 05:54:15,477 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-13 05:54:15,487 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-13 05:54:15,508 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=169, AvailableMemoryMB=4999 2023-03-13 05:54:15,510 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-13 05:54:15,510 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-13 05:54:15,677 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013503065:84 , srv1015069263:84 , srv1018834650:84 , srv1056665237:84 , srv1081092889:84 , srv1145413836:84 , srv1177236135:84 , srv1211272759:84 , srv1235028085:84 , srv1235109536:84 , srv1254849546:84 , srv1259270384:84 , srv126385004:84 , srv1268283659:84 , srv1274662540:84 , srv1274977056:84 , srv1281539201:84 , srv1285820360:84 , srv1305045537:84 , srv1305448933:84 , srv1313437456:84 , srv1322232062:84 , srv1346498321:84 , srv1353324259:84 , srv13694415:84 , srv1381857264:84 , srv1407401617:84 , srv1412359541:84 , srv1422844524:84 , srv1475415918:84 , srv1507761364:84 , srv1528393343:84 , srv156380794:84 , srv1565722100:84 , srv1571654991:84 , srv1573041646:84 , srv1596928000:84 , srv1608205755:84 , srv1669384421:84 , srv1698396592:84 , srv1704128700:84 , srv1711959572:84 , srv1758649571:84 , srv1765624223:84 , srv1768594702:84 , srv1856789123:84 , srv1875598162:84 , srv1883780326:84 , srv1909580090:84 , srv1913825390:84 , srv1914115952:84 , srv1914272598:84 , srv1952531409:84 , srv198646199:84 , srv1991560656:84 , srv2039197820:84 , srv2069547647:84 , srv2070052270:84 , srv2076293545:84 , srv2077944946:84 , srv2096047303:84 , srv2096121240:84 , srv21745707:84 , srv266657210:84 , srv281219551:84 , srv305356003:84 , srv328887566:84 , srv342357379:84 , srv349110310:84 , srv395075686:84 , srv403192743:84 , srv433970295:84 , srv458334075:84 , srv465170103:84 , srv466182630:84 , srv533799733:84 , srv555982255:84 , srv58844553:84 , srv601403433:84 , srv604883999:84 , srv609981670:84 , srv63115446:84 , srv638556145:84 , srv662565530:84 , srv669813071:84 , srv73687932:84 , srv738715067:84 , srv764502465:84 , srv770424170:84 , srv822217691:84 , srv828849928:84 , srv871526766:84 , srv874817688:84 , srv894756696:84 , srv929826648:84 , srv929946587:84 , srv930578658:84 , srv935833648:84 , srv99298156:84 , srv1148998135:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-13 05:54:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:54:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2069547647=57, srv929826648=95, srv1711959572=42, srv1268283659=14, srv305356003=66, srv1381857264=26, srv662565530=84, srv1015069263=1, srv1056665237=3, srv1573041646=36, srv669813071=85, srv1698396592=40, srv466182630=75, srv828849928=91, srv1274662540=15, srv935833648=98, srv1259270384=12, srv609981670=81, srv874817688=93, srv533799733=76, srv1235028085=9, srv1596928000=37, srv1909580090=49, srv2077944946=60, srv2070052270=58, srv738715067=87, srv1148998135=6, srv2076293545=59, srv328887566=67, srv1913825390=50, srv266657210=64, srv1758649571=43, srv58844553=78, srv1914115952=51, srv1914272598=52, srv1528393343=32, srv1669384421=39, srv764502465=88, srv1856789123=46, srv555982255=77, srv1081092889=4, srv1018834650=2, srv1571654991=35, srv2096121240=62, srv1475415918=30, srv1281539201=17, srv1305045537=19, srv458334075=73, srv126385004=13, srv638556145=83, srv1346498321=23, srv281219551=65, srv1013503065=0, srv894756696=94, srv1211272759=8, srv604883999=80, srv1883780326=48, srv2096047303=61, srv1313437456=21, srv1353324259=24, srv1305448933=20, srv1274977056=16, srv1765624223=44, srv770424170=89, srv21745707=63, srv1768594702=45, srv1285820360=18, srv63115446=82, srv156380794=33, srv433970295=72, srv1412359541=28, srv1565722100=34, srv99298156=99, srv465170103=74, srv1177236135=7, srv13694415=25, srv1507761364=31, srv1407401617=27, srv930578658=97, srv1422844524=29, srv1704128700=41, srv403192743=71, srv73687932=86, srv342357379=68, srv1991560656=55, srv1952531409=53, srv822217691=90, srv1608205755=38, srv198646199=54, srv1254849546=11, srv395075686=70, srv1235109536=10, srv1322232062=22, srv1145413836=5, srv2039197820=56, srv349110310=69, srv601403433=79, srv871526766=92, srv929946587=96, srv1875598162=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:54:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:54:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:54:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:54:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:54:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:54:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:54:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:54:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:54:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:54:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:54:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:54:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:54:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:54:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:54:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:54:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:54:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:54:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:54:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:54:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:54:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-13 05:54:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-13 05:54:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-13 05:54:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-13 05:54:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-13 05:54:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-13 05:54:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-13 05:54:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-13 05:54:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-13 05:54:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-13 05:54:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-13 05:54:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-13 05:54:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-13 05:54:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-13 05:54:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-13 05:54:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-13 05:54:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-13 05:54:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-13 05:54:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-13 05:54:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-13 05:54:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-13 05:54:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-13 05:54:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-13 05:54:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-13 05:54:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-13 05:54:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-13 05:54:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-13 05:54:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-13 05:54:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-13 05:54:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-13 05:54:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-13 05:54:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-13 05:54:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-13 05:54:16,042 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:16,042 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-13 05:54:16,042 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-13 05:54:16,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 547 ms to try 5000 different iterations. Found a solution that moves 4700 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1804397859290845. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04094469913895446, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06096493076670759, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5222222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04094469913895446, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:16,408 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv458334075:81 , srv1013503065:82 , srv1353324259:82 , srv1883780326:82 , srv466182630:82 , srv1313437456:83 , srv1412359541:83 , srv1475415918:83 , srv198646199:83 , srv2039197820:83 , srv2069547647:83 , srv349110310:83 , srv638556145:83 , srv1081092889:84 , srv1145413836:84 , srv1254849546:84 , srv126385004:84 , srv1268283659:84 , srv1914272598:84 , srv266657210:84 , srv403192743:84 , srv555982255:84 , srv601403433:84 , srv764502465:84 , srv770424170:84 , srv828849928:84 , srv1056665237:85 , srv1281539201:85 , srv13694415:85 , srv1422844524:85 , srv1596928000:85 , srv1704128700:85 , srv1765624223:85 , srv1875598162:85 , srv1909580090:85 , srv2096047303:85 , srv871526766:85 , srv1015069263:86 , srv1235109536:86 , srv1259270384:86 , srv1274977056:86 , srv1305045537:86 , srv1305448933:86 , srv1346498321:86 , srv1381857264:86 , srv1407401617:86 , srv2077944946:86 , srv2096121240:86 , srv305356003:86 , srv609981670:86 , srv63115446:86 , srv738715067:86 , srv822217691:86 , srv894756696:86 , srv929946587:86 , srv1235028085:87 , srv1528393343:87 , srv1758649571:87 , srv1913825390:87 , srv1991560656:87 , srv21745707:87 , srv395075686:87 , srv433970295:87 , srv930578658:87 , srv1177236135:88 , srv1274662540:88 , srv1285820360:88 , srv1322232062:88 , srv1507761364:88 , srv156380794:88 , srv1565722100:88 , srv1669384421:88 , srv1698396592:88 , srv1711959572:88 , srv1768594702:88 , srv1914115952:88 , srv1952531409:88 , srv2070052270:88 , srv281219551:88 , srv533799733:88 , srv73687932:88 , srv874817688:88 , srv1018834650:89 , srv1211272759:89 , srv1573041646:89 , srv1608205755:89 , srv342357379:89 , srv465170103:89 , srv604883999:89 , srv1571654991:90 , srv1856789123:90 , srv2076293545:90 , srv328887566:90 , srv935833648:90 , srv99298156:90 , srv662565530:91 , srv929826648:91 , srv58844553:93 , srv669813071:93 , srv1148998135:454 } 2023-03-13 05:54:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:54:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2069547647=57, srv929826648=95, srv1711959572=42, srv1268283659=14, srv305356003=66, srv1381857264=26, srv662565530=84, srv1015069263=1, srv1056665237=3, srv1573041646=36, srv669813071=85, srv1698396592=40, srv466182630=75, srv828849928=91, srv1274662540=15, srv935833648=98, srv1259270384=12, srv609981670=81, srv874817688=93, srv533799733=76, srv1235028085=9, srv1596928000=37, srv1909580090=49, srv2077944946=60, srv2070052270=58, srv738715067=87, srv1148998135=6, srv2076293545=59, srv328887566=67, srv1913825390=50, srv266657210=64, srv1758649571=43, srv58844553=78, srv1914115952=51, srv1914272598=52, srv1528393343=32, srv1669384421=39, srv764502465=88, srv1856789123=46, srv555982255=77, srv1081092889=4, srv1018834650=2, srv1571654991=35, srv2096121240=62, srv1475415918=30, srv1281539201=17, srv1305045537=19, srv458334075=73, srv126385004=13, srv638556145=83, srv1346498321=23, srv281219551=65, srv1013503065=0, srv894756696=94, srv1211272759=8, srv604883999=80, srv1883780326=48, srv2096047303=61, srv1313437456=21, srv1353324259=24, srv1305448933=20, srv1274977056=16, srv1765624223=44, srv770424170=89, srv21745707=63, srv1768594702=45, srv1285820360=18, srv63115446=82, srv156380794=33, srv433970295=72, srv1412359541=28, srv1565722100=34, srv99298156=99, srv465170103=74, srv1177236135=7, srv13694415=25, srv1507761364=31, srv1407401617=27, srv930578658=97, srv1422844524=29, srv1704128700=41, srv403192743=71, srv73687932=86, srv342357379=68, srv1991560656=55, srv1952531409=53, srv822217691=90, srv1608205755=38, srv198646199=54, srv1254849546=11, srv395075686=70, srv1235109536=10, srv1322232062=22, srv1145413836=5, srv2039197820=56, srv349110310=69, srv601403433=79, srv871526766=92, srv929946587=96, srv1875598162=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:54:16,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:54:16,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-13 05:54:16,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-13 05:54:16,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-13 05:54:16,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-13 05:54:16,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-13 05:54:16,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-13 05:54:16,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-13 05:54:16,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04094469913895446, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06096493076670759, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04094469913895446, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:16,578 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-13 05:54:16,578 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18040687582568274, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04094469913895446, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06096493076670759, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04094469913895446, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-13 05:54:16,912 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 446 ms to try 5000 different iterations. Found a solution that moves 2625 regions; Going from a computed imbalance of 0.18040687582568274 to a new imbalance of 0.013600212119816605. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023733146697206862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03519705629972634, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023733146697206862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01025); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:16,921 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv58844553:77 , srv1211272759:81 , srv2076293545:81 , srv662565530:81 , srv669813071:81 , srv1177236135:82 , srv1571654991:82 , srv328887566:82 , srv604883999:82 , srv929826648:82 , srv1235028085:83 , srv1565722100:83 , srv1573041646:83 , srv1711959572:83 , srv1758649571:83 , srv1768594702:83 , srv1856789123:83 , srv2070052270:83 , srv465170103:83 , srv73687932:83 , srv738715067:83 , srv1018834650:84 , srv1274977056:84 , srv1285820360:84 , srv1305448933:84 , srv1322232062:84 , srv13694415:84 , srv1381857264:84 , srv1407401617:84 , srv1507761364:84 , srv156380794:84 , srv1608205755:84 , srv1698396592:84 , srv2096121240:84 , srv281219551:84 , srv342357379:84 , srv395075686:84 , srv533799733:84 , srv63115446:84 , srv874817688:84 , srv935833648:84 , srv1056665237:85 , srv1281539201:85 , srv1528393343:85 , srv1913825390:85 , srv1952531409:85 , srv21745707:85 , srv305356003:85 , srv609981670:85 , srv822217691:85 , srv894756696:85 , srv930578658:85 , srv99298156:85 , srv1254849546:86 , srv1259270384:86 , srv1274662540:86 , srv1422844524:86 , srv1596928000:86 , srv1704128700:86 , srv1875598162:86 , srv1914115952:86 , srv1991560656:86 , srv433970295:86 , srv871526766:86 , srv126385004:87 , srv1305045537:87 , srv1346498321:87 , srv1669384421:87 , srv1883780326:87 , srv2077944946:87 , srv403192743:87 , srv601403433:87 , srv929946587:87 , srv1015069263:88 , srv1235109536:88 , srv1268283659:88 , srv1313437456:88 , srv1353324259:88 , srv1412359541:88 , srv1914272598:88 , srv2096047303:88 , srv638556145:88 , srv764502465:88 , srv1081092889:89 , srv1145413836:89 , srv1909580090:89 , srv198646199:89 , srv266657210:89 , srv349110310:89 , srv466182630:89 , srv770424170:89 , srv828849928:89 , srv1475415918:90 , srv1765624223:90 , srv2039197820:90 , srv555982255:90 , srv2069547647:91 , srv1013503065:92 , srv458334075:92 , srv1148998135:531 } 2023-03-13 05:54:16,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:54:16,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2069547647=57, srv929826648=95, srv1711959572=42, srv1268283659=14, srv305356003=66, srv1381857264=26, srv662565530=84, srv1015069263=1, srv1056665237=3, srv1573041646=36, srv669813071=85, srv1698396592=40, srv466182630=75, srv828849928=91, srv1274662540=15, srv935833648=98, srv1259270384=12, srv609981670=81, srv874817688=93, srv533799733=76, srv1235028085=9, srv1596928000=37, srv1909580090=49, srv2077944946=60, srv2070052270=58, srv738715067=87, srv1148998135=6, srv2076293545=59, srv328887566=67, srv1913825390=50, srv266657210=64, srv1758649571=43, srv58844553=78, srv1914115952=51, srv1914272598=52, srv1528393343=32, srv1669384421=39, srv764502465=88, srv1856789123=46, srv555982255=77, srv1081092889=4, srv1018834650=2, srv1571654991=35, srv2096121240=62, srv1475415918=30, srv1281539201=17, srv1305045537=19, srv458334075=73, srv126385004=13, srv638556145=83, srv1346498321=23, srv281219551=65, srv1013503065=0, srv894756696=94, srv1211272759=8, srv604883999=80, srv1883780326=48, srv2096047303=61, srv1313437456=21, srv1353324259=24, srv1305448933=20, srv1274977056=16, srv1765624223=44, srv770424170=89, srv21745707=63, srv1768594702=45, srv1285820360=18, srv63115446=82, srv156380794=33, srv433970295=72, srv1412359541=28, srv1565722100=34, srv99298156=99, srv465170103=74, srv1177236135=7, srv13694415=25, srv1507761364=31, srv1407401617=27, srv930578658=97, srv1422844524=29, srv1704128700=41, srv403192743=71, srv73687932=86, srv342357379=68, srv1991560656=55, srv1952531409=53, srv822217691=90, srv1608205755=38, srv198646199=54, srv1254849546=11, srv395075686=70, srv1235109536=10, srv1322232062=22, srv1145413836=5, srv2039197820=56, srv349110310=69, srv601403433=79, srv871526766=92, srv929946587=96, srv1875598162=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:54:16,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:54:16,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:54:16,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:54:16,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-13 05:54:16,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-13 05:54:16,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-13 05:54:16,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-13 05:54:16,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-13 05:54:16,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-13 05:54:16,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-13 05:54:16,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-13 05:54:17,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023733146697206862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03519705629972634, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023733146697206862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01025); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:17,060 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-13 05:54:17,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013581831476959245, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023733146697206862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03519705629972634, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023733146697206862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01025); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-13 05:54:17,345 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 387 ms to try 5000 different iterations. Found a solution that moves 1016 regions; Going from a computed imbalance of 0.013581831476959245 to a new imbalance of 7.79443048518202E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005445601013231794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007415820138576829); MoveCostFunction : (multiplier=7.0, imbalance=0.11288888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005445601013231794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:17,347 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv99298156:82 , srv1235109536:83 , srv1274662540:83 , srv1669384421:83 , srv1765624223:83 , srv1909580090:83 , srv828849928:83 , srv1013503065:84 , srv1145413836:84 , srv1856789123:84 , srv1914115952:84 , srv198646199:84 , srv2039197820:84 , srv2069547647:84 , srv266657210:84 , srv555982255:84 , srv929826648:84 , srv929946587:84 , srv935833648:84 , srv1015069263:85 , srv1081092889:85 , srv1259270384:85 , srv1305045537:85 , srv1322232062:85 , srv1346498321:85 , srv1475415918:85 , srv1507761364:85 , srv156380794:85 , srv1571654991:85 , srv1608205755:85 , srv1698396592:85 , srv1711959572:85 , srv1913825390:85 , srv1952531409:85 , srv2070052270:85 , srv2077944946:85 , srv2096047303:85 , srv342357379:85 , srv349110310:85 , srv433970295:85 , srv458334075:85 , srv465170103:85 , srv533799733:85 , srv669813071:85 , srv764502465:85 , srv770424170:85 , srv871526766:85 , srv930578658:85 , srv1018834650:86 , srv1268283659:86 , srv1285820360:86 , srv1313437456:86 , srv1412359541:86 , srv1528393343:86 , srv1565722100:86 , srv1573041646:86 , srv1704128700:86 , srv1875598162:86 , srv1991560656:86 , srv2076293545:86 , srv21745707:86 , srv281219551:86 , srv328887566:86 , srv395075686:86 , srv601403433:86 , srv604883999:86 , srv609981670:86 , srv638556145:86 , srv662565530:86 , srv73687932:86 , srv874817688:86 , srv894756696:86 , srv1056665237:87 , srv1254849546:87 , srv126385004:87 , srv1274977056:87 , srv1281539201:87 , srv1305448933:87 , srv1353324259:87 , srv1381857264:87 , srv1422844524:87 , srv1758649571:87 , srv1914272598:87 , srv2096121240:87 , srv305356003:87 , srv466182630:87 , srv822217691:87 , srv1177236135:88 , srv1211272759:88 , srv1235028085:88 , srv13694415:88 , srv1407401617:88 , srv1596928000:88 , srv1768594702:88 , srv403192743:88 , srv58844553:88 , srv63115446:88 , srv738715067:88 , srv1883780326:89 , srv1148998135:521 } 2023-03-13 05:54:17,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:54:17,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2069547647=57, srv929826648=95, srv1711959572=42, srv1268283659=14, srv305356003=66, srv1381857264=26, srv662565530=84, srv1015069263=1, srv1056665237=3, srv1573041646=36, srv669813071=85, srv1698396592=40, srv466182630=75, srv828849928=91, srv1274662540=15, srv935833648=98, srv1259270384=12, srv609981670=81, srv874817688=93, srv533799733=76, srv1235028085=9, srv1596928000=37, srv1909580090=49, srv2077944946=60, srv2070052270=58, srv738715067=87, srv1148998135=6, srv2076293545=59, srv328887566=67, srv1913825390=50, srv266657210=64, srv1758649571=43, srv58844553=78, srv1914115952=51, srv1914272598=52, srv1528393343=32, srv1669384421=39, srv764502465=88, srv1856789123=46, srv555982255=77, srv1081092889=4, srv1018834650=2, srv1571654991=35, srv2096121240=62, srv1475415918=30, srv1281539201=17, srv1305045537=19, srv458334075=73, srv126385004=13, srv638556145=83, srv1346498321=23, srv281219551=65, srv1013503065=0, srv894756696=94, srv1211272759=8, srv604883999=80, srv1883780326=48, srv2096047303=61, srv1313437456=21, srv1353324259=24, srv1305448933=20, srv1274977056=16, srv1765624223=44, srv770424170=89, srv21745707=63, srv1768594702=45, srv1285820360=18, srv63115446=82, srv156380794=33, srv433970295=72, srv1412359541=28, srv1565722100=34, srv99298156=99, srv465170103=74, srv1177236135=7, srv13694415=25, srv1507761364=31, srv1407401617=27, srv930578658=97, srv1422844524=29, srv1704128700=41, srv403192743=71, srv73687932=86, srv342357379=68, srv1991560656=55, srv1952531409=53, srv822217691=90, srv1608205755=38, srv198646199=54, srv1254849546=11, srv395075686=70, srv1235109536=10, srv1322232062=22, srv1145413836=5, srv2039197820=56, srv349110310=69, srv601403433=79, srv871526766=92, srv929946587=96, srv1875598162=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:54:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:54:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:54:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:54:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-13 05:54:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-13 05:54:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-13 05:54:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-13 05:54:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-13 05:54:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-13 05:54:17,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-13 05:54:17,493 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005445601013231794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007415820138576829); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005445601013231794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:17,493 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-13 05:54:17,494 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.723288644636971E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005445601013231794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007415820138576829); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005445601013231794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-13 05:54:17,768 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations. Found a solution that moves 250 regions; Going from a computed imbalance of 7.723288644636971E-4 to a new imbalance of 1.9917989237262437E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.027777777777777776, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:17,769 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018834650:83 , srv1268283659:83 , srv1768594702:83 , srv1914272598:83 , srv2096047303:83 , srv305356003:83 , srv403192743:83 , srv63115446:83 , srv669813071:83 , srv1013503065:84 , srv1015069263:84 , srv1211272759:84 , srv1235028085:84 , srv1235109536:84 , srv126385004:84 , srv1285820360:84 , srv1305045537:84 , srv1322232062:84 , srv1346498321:84 , srv1407401617:84 , srv1475415918:84 , srv1528393343:84 , srv1565722100:84 , srv1596928000:84 , srv1608205755:84 , srv1669384421:84 , srv1765624223:84 , srv1883780326:84 , srv1914115952:84 , srv1952531409:84 , srv1991560656:84 , srv2077944946:84 , srv21745707:84 , srv281219551:84 , srv328887566:84 , srv433970295:84 , srv458334075:84 , srv466182630:84 , srv555982255:84 , srv58844553:84 , srv638556145:84 , srv662565530:84 , srv770424170:84 , srv822217691:84 , srv935833648:84 , srv1056665237:85 , srv1081092889:85 , srv1145413836:85 , srv1177236135:85 , srv1254849546:85 , srv1259270384:85 , srv1274662540:85 , srv1274977056:85 , srv1281539201:85 , srv1305448933:85 , srv1313437456:85 , srv1353324259:85 , srv13694415:85 , srv1381857264:85 , srv1412359541:85 , srv1422844524:85 , srv1507761364:85 , srv156380794:85 , srv1571654991:85 , srv1573041646:85 , srv1698396592:85 , srv1704128700:85 , srv1711959572:85 , srv1758649571:85 , srv1856789123:85 , srv1875598162:85 , srv1909580090:85 , srv1913825390:85 , srv2039197820:85 , srv2069547647:85 , srv2076293545:85 , srv2096121240:85 , srv266657210:85 , srv342357379:85 , srv349110310:85 , srv395075686:85 , srv465170103:85 , srv533799733:85 , srv601403433:85 , srv604883999:85 , srv609981670:85 , srv73687932:85 , srv738715067:85 , srv764502465:85 , srv874817688:85 , srv894756696:85 , srv929826648:85 , srv929946587:85 , srv930578658:85 , srv99298156:85 , srv198646199:86 , srv2070052270:86 , srv828849928:86 , srv871526766:86 , srv1148998135:635 } 2023-03-13 05:54:17,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:54:17,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2069547647=57, srv929826648=95, srv1711959572=42, srv1268283659=14, srv305356003=66, srv1381857264=26, srv662565530=84, srv1015069263=1, srv1056665237=3, srv1573041646=36, srv669813071=85, srv1698396592=40, srv466182630=75, srv828849928=91, srv1274662540=15, srv935833648=98, srv1259270384=12, srv609981670=81, srv874817688=93, srv533799733=76, srv1235028085=9, srv1596928000=37, srv1909580090=49, srv2077944946=60, srv2070052270=58, srv738715067=87, srv1148998135=6, srv2076293545=59, srv328887566=67, srv1913825390=50, srv266657210=64, srv1758649571=43, srv58844553=78, srv1914115952=51, srv1914272598=52, srv1528393343=32, srv1669384421=39, srv764502465=88, srv1856789123=46, srv555982255=77, srv1081092889=4, srv1018834650=2, srv1571654991=35, srv2096121240=62, srv1475415918=30, srv1281539201=17, srv1305045537=19, srv458334075=73, srv126385004=13, srv638556145=83, srv1346498321=23, srv281219551=65, srv1013503065=0, srv894756696=94, srv1211272759=8, srv604883999=80, srv1883780326=48, srv2096047303=61, srv1313437456=21, srv1353324259=24, srv1305448933=20, srv1274977056=16, srv1765624223=44, srv770424170=89, srv21745707=63, srv1768594702=45, srv1285820360=18, srv63115446=82, srv156380794=33, srv433970295=72, srv1412359541=28, srv1565722100=34, srv99298156=99, srv465170103=74, srv1177236135=7, srv13694415=25, srv1507761364=31, srv1407401617=27, srv930578658=97, srv1422844524=29, srv1704128700=41, srv403192743=71, srv73687932=86, srv342357379=68, srv1991560656=55, srv1952531409=53, srv822217691=90, srv1608205755=38, srv198646199=54, srv1254849546=11, srv395075686=70, srv1235109536=10, srv1322232062=22, srv1145413836=5, srv2039197820=56, srv349110310=69, srv601403433=79, srv871526766=92, srv929946587=96, srv1875598162=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:54:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-13 05:54:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-13 05:54:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-13 05:54:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-13 05:54:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-13 05:54:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-13 05:54:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-13 05:54:17,890 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.816745182227604E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:17,891 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1018834650:83 , srv1268283659:83 , srv1768594702:83 , srv1914272598:83 , srv2096047303:83 , srv305356003:83 , srv403192743:83 , srv63115446:83 , srv669813071:83 , srv1013503065:84 , srv1015069263:84 , srv1211272759:84 , srv1235028085:84 , srv1235109536:84 , srv126385004:84 , srv1285820360:84 , srv1305045537:84 , srv1322232062:84 , srv1346498321:84 , srv1407401617:84 , srv1475415918:84 , srv1528393343:84 , srv1565722100:84 , srv1596928000:84 , srv1608205755:84 , srv1669384421:84 , srv1765624223:84 , srv1883780326:84 , srv1914115952:84 , srv1952531409:84 , srv1991560656:84 , srv2077944946:84 , srv21745707:84 , srv281219551:84 , srv328887566:84 , srv433970295:84 , srv458334075:84 , srv466182630:84 , srv555982255:84 , srv58844553:84 , srv638556145:84 , srv662565530:84 , srv770424170:84 , srv822217691:84 , srv935833648:84 , srv1056665237:85 , srv1081092889:85 , srv1145413836:85 , srv1177236135:85 , srv1254849546:85 , srv1259270384:85 , srv1274662540:85 , srv1274977056:85 , srv1281539201:85 , srv1305448933:85 , srv1313437456:85 , srv1353324259:85 , srv13694415:85 , srv1381857264:85 , srv1412359541:85 , srv1422844524:85 , srv1507761364:85 , srv156380794:85 , srv1571654991:85 , srv1573041646:85 , srv1698396592:85 , srv1704128700:85 , srv1711959572:85 , srv1758649571:85 , srv1856789123:85 , srv1875598162:85 , srv1909580090:85 , srv1913825390:85 , srv2039197820:85 , srv2069547647:85 , srv2076293545:85 , srv2096121240:85 , srv266657210:85 , srv342357379:85 , srv349110310:85 , srv395075686:85 , srv465170103:85 , srv533799733:85 , srv601403433:85 , srv604883999:85 , srv609981670:85 , srv73687932:85 , srv738715067:85 , srv764502465:85 , srv874817688:85 , srv894756696:85 , srv929826648:85 , srv929946587:85 , srv930578658:85 , srv99298156:85 , srv198646199:86 , srv2070052270:86 , srv828849928:86 , srv871526766:86 , srv1148998135:635 } 2023-03-13 05:54:17,917 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 246), ProcessCount=169 (was 169), AvailableMemoryMB=4471 (was 4999) 2023-03-13 05:54:17,924 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=169, AvailableMemoryMB=4470 2023-03-13 05:54:17,925 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-13 05:54:17,925 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-13 05:54:17,935 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1070401206:84 , srv1106097369:84 , srv1141804888:84 , srv1208738231:84 , srv1240285086:84 , srv1316082748:84 , srv1464847951:84 , srv1525848498:84 , srv1568637286:84 , srv1734590660:84 , srv1794807776:84 , srv1795745928:84 , srv1886313965:84 , srv1887349860:84 , srv1998957609:84 , srv2058132701:84 , srv2086953280:84 , srv2120123919:84 , srv276948502:84 , srv290958190:84 , srv38250233:84 , srv419586329:84 , srv443839434:84 , srv481477860:84 , srv516879035:84 , srv518883272:84 , srv57325226:84 , srv748584483:84 , srv996634143:84 , srv770109182:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-13 05:54:17,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:54:17,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1795745928=11, srv996634143=29, srv1240285086=4, srv38250233=20, srv2120123919=17, srv1734590660=9, srv2058132701=15, srv419586329=21, srv2086953280=16, srv1886313965=12, srv1070401206=0, srv1316082748=5, srv276948502=18, srv481477860=23, srv518883272=25, srv57325226=26, srv748584483=27, srv290958190=19, srv443839434=22, srv770109182=28, srv1464847951=6, srv1525848498=7, srv1208738231=3, srv1141804888=2, srv1998957609=14, srv1568637286=8, srv1794807776=10, srv1887349860=13, srv1106097369=1, srv516879035=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:54:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:54:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:54:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:54:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:54:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:54:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:54:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-13 05:54:17,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1034489986620521, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:54:17,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912519007681144, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1034489986620521, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-13 05:56:17,951 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6204829 different iterations. Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912519007681144 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:56:17,957 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1070401206:90 , srv1106097369:90 , srv1141804888:90 , srv1208738231:90 , srv1240285086:90 , srv1316082748:90 , srv1464847951:90 , srv1525848498:90 , srv1568637286:90 , srv1734590660:90 , srv1794807776:90 , srv1795745928:90 , srv1886313965:90 , srv1887349860:90 , srv1998957609:90 , srv2058132701:90 , srv2086953280:90 , srv2120123919:90 , srv276948502:90 , srv290958190:90 , srv38250233:90 , srv419586329:90 , srv443839434:90 , srv481477860:90 , srv516879035:90 , srv518883272:90 , srv57325226:90 , srv748584483:90 , srv770109182:90 , srv996634143:90 } 2023-03-13 05:56:17,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-13 05:56:17,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1795745928=11, srv996634143=29, srv1240285086=4, srv38250233=20, srv2120123919=17, srv1734590660=9, srv2058132701=15, srv419586329=21, srv2086953280=16, srv1886313965=12, srv1070401206=0, srv1316082748=5, srv276948502=18, srv481477860=23, srv518883272=25, srv57325226=26, srv748584483=27, srv290958190=19, srv443839434=22, srv770109182=28, srv1464847951=6, srv1525848498=7, srv1208738231=3, srv1141804888=2, srv1998957609=14, srv1568637286=8, srv1794807776=10, srv1887349860=13, srv1106097369=1, srv516879035=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-13 05:56:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-13 05:56:17,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-13 05:56:17,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-13 05:56:17,976 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1070401206:90 , srv1106097369:90 , srv1141804888:90 , srv1208738231:90 , srv1240285086:90 , srv1316082748:90 , srv1464847951:90 , srv1525848498:90 , srv1568637286:90 , srv1734590660:90 , srv1794807776:90 , srv1795745928:90 , srv1886313965:90 , srv1887349860:90 , srv1998957609:90 , srv2058132701:90 , srv2086953280:90 , srv2120123919:90 , srv276948502:90 , srv290958190:90 , srv38250233:90 , srv419586329:90 , srv443839434:90 , srv481477860:90 , srv516879035:90 , srv518883272:90 , srv57325226:90 , srv748584483:90 , srv770109182:90 , srv996634143:90 } 2023-03-13 05:56:17,988 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=141 (was 246), ProcessCount=169 (was 169), AvailableMemoryMB=4189 (was 4470)